RDMA/hns: Create QP with selected QPN for bank load balance
Subsystems (2)
PATCH v2 1/1 RDMA/hns: Create QP with selected QPN for bank load balance
From: Yangyang Li <[email protected]>
In order to improve performance by balancing the load between different banks of cache, the QPC cache is desigend to choose one of 8 banks according to lower 3 bits of QPN. The hns driver needs to count the number of QP on each bank and then assigns the QP being created to the bank with the minimum load first.
---
Changes since v1:
- Use ida instead of bitmap to allocate QPN.
- Remove CQ parts, which will be reorganized in a separate patch.
drivers/infiniband/hw/hns/hns_roce_device.h | 15 ++++- drivers/infiniband/hw/hns/hns_roce_qp.c | 100 +++++++++++++++++++++++----- 2 files changed, 96 insertions(+), 19 deletions(-)
drivers/infiniband/hw/hns/hns_roce_device.h +13 -2
@@ -117,6 +117,8 @@#define HNS_ROCE_IDX_QUE_ENTRY_SZ 4#define SRQ_DB_REG 0x230 +#define HNS_ROCE_QP_BANK_NUM 8 +/* The chip implementation of the consumer index is calculated * according to twice the actual EQ depth */
@@ -512,13 +514,22 @@ struct hns_roce_uar_table {struct hns_roce_bitmap bitmap; }; +struct hns_roce_bank { +struct ida ida; +u32 inuse;/* Number of IDs allocated */ +u32 min;/* Lowest ID to allocate. */ +u32 max;/* Highest ID to allocate. */ +u32 rsv_bot;/* Reserved bottom ids */ +}; +struct hns_roce_qp_table { -struct hns_roce_bitmap bitmap;struct hns_roce_hem_table qp_table;struct hns_roce_hem_table irrl_table;struct hns_roce_hem_table trrl_table;struct hns_roce_hem_table sccc_table;struct mutex scc_mutex; +struct hns_roce_bank bank[HNS_ROCE_QP_BANK_NUM ]; +spinlock_t bank_lock; };struct hns_roce_cq_table {
@@ -768,7 +779,7 @@ struct hns_roce_caps {u32 max_rq_sg;u32 max_extend_sg;int num_qps; -int reserved_qps; +u32 reserved_qps;int num_qpc_timer;int num_cqc_timer;int num_srqs;
drivers/infiniband/hw/hns/hns_roce_qp.c +83 -17
148 lines not on this page · raw message
Signature · 1 line
2.8.1
drivers/infiniband/hw/hns/hns_roce_qp.c static void hns_roce_ib_qp_event(struct hns_roce_qp *hr_qp, · line 183
Quoted 9 lines · Weihang Li
drivers/infiniband/hw/hns/hns_roce_qp.c +9
-0+static int alloc_qpn_with_bankid (struct hns_roce_bank *bank,u8 bankid, +unsigned long *qpn) +{ +int id; + + id =ida_alloc_range (&bank->ida, bank->min, bank->max,GFP_KERNEL ); +if (id <0 ) { + id =ida_alloc_range (&bank->ida, bank->rsv_bot, bank->min, +GFP_KERNEL );
Shouldn't this one be bank->max not min? That is the usual way to write a cyclic allocator over this kind of API. See the logic in __xa_alloc_cyclic()
Jason
Quoted 14 lines · Jason Gunthorpe
drivers/infiniband/hw/hns/hns_roce_qp.c static void hns_roce_ib_qp_event(struct hns_roce_qp *hr_qp, · line 183
Quoted 9 lines · Weihang Li
drivers/infiniband/hw/hns/hns_roce_qp.c +9
-0+static int alloc_qpn_with_bankid (struct hns_roce_bank *bank,u8 bankid, +unsigned long *qpn) +{ +int id; + + id =ida_alloc_range (&bank->ida, bank->min, bank->max,GFP_KERNEL ); +if (id <0 ) { + id =ida_alloc_range (&bank->ida, bank->rsv_bot, bank->min, +GFP_KERNEL );Shouldn't this one be
bank->maxnot min? That is the usual way to write a cyclic allocator over this kind of API. See the logic in__xa_alloc_cyclic()
Quoted 2 lines
Jason
Thanks for your advice, it's clearer to use bank->max here. We will refer to how __xa_alloc_cyclic() is implemented.
Weihang