Skip to contentSkip to conversation

RDMA/hns: Create QP with selected QPN for bank load balance

PATCH Superseded Weihang Li 2 participants 3 messages Nov 23, 2020 – Nov 24, 2020

Subsystems (2)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Weihang Li

author
#0 lore raw

PATCH v2 1/1 RDMA/hns: Create QP with selected QPN for bank load balance

From: Yangyang Li <[email protected]>

In order to improve performance by balancing the load between different banks of cache, the QPC cache is desigend to choose one of 8 banks according to lower 3 bits of QPN. The hns driver needs to count the number of QP on each bank and then assigns the QP being created to the bank with the minimum load first.

---
Changes since v1:
- Use ida instead of bitmap to allocate QPN.
- Remove CQ parts, which will be reorganized in a separate patch.

 drivers/infiniband/hw/hns/hns_roce_device.h |  15 ++++-
 drivers/infiniband/hw/hns/hns_roce_qp.c     | 100 +++++++++++++++++++++++-----
 2 files changed, 96 insertions(+), 19 deletions(-)
drivers/infiniband/hw/hns/hns_roce_device.h +13 -2
index 1d99022..e9783cc 100644
@@ -117,6 +117,8 @@
 #define HNS_ROCE_IDX_QUE_ENTRY_SZ		4
 #define SRQ_DB_REG				0x230
 
+#define HNS_ROCE_QP_BANK_NUM 8
+
 /* The chip implementation of the consumer index is calculated
  * according to twice the actual EQ depth
  */
@@ -512,13 +514,22 @@ struct hns_roce_uar_table {
 	struct hns_roce_bitmap bitmap;
 };
 
+struct hns_roce_bank {
+	struct ida ida;
+	u32 inuse; /* Number of IDs allocated */
+	u32 min; /* Lowest ID to allocate.  */
+	u32 max; /* Highest ID to allocate. */
+	u32 rsv_bot; /* Reserved bottom ids */
+};
+
 struct hns_roce_qp_table {
-	struct hns_roce_bitmap		bitmap;
 	struct hns_roce_hem_table	qp_table;
 	struct hns_roce_hem_table	irrl_table;
 	struct hns_roce_hem_table	trrl_table;
 	struct hns_roce_hem_table	sccc_table;
 	struct mutex			scc_mutex;
+	struct hns_roce_bank bank[HNS_ROCE_QP_BANK_NUM];
+	spinlock_t bank_lock;
 };
 
 struct hns_roce_cq_table {
@@ -768,7 +779,7 @@ struct hns_roce_caps {
 	u32		max_rq_sg;
 	u32		max_extend_sg;
 	int		num_qps;
-	int             reserved_qps;
+	u32             reserved_qps;
 	int		num_qpc_timer;
 	int		num_cqc_timer;
 	int		num_srqs;
drivers/infiniband/hw/hns/hns_roce_qp.c +83 -17

148 lines not on this page · raw message

Signature · 1 line
2.8.1
#1 lore raw

drivers/infiniband/hw/hns/hns_roce_qp.c static void hns_roce_ib_qp_event(struct hns_roce_qp *hr_qp, · line 183

Quoted 9 lines · Weihang Li
drivers/infiniband/hw/hns/hns_roce_qp.c +9 -0
+static int alloc_qpn_with_bankid(struct hns_roce_bank *bank, u8 bankid,
+				 unsigned long *qpn)
+{
+	int id;
+
+	id = ida_alloc_range(&bank->ida, bank->min, bank->max, GFP_KERNEL);
+	if (id < 0) {
+		id = ida_alloc_range(&bank->ida, bank->rsv_bot, bank->min,
+				     GFP_KERNEL);

Shouldn't this one be bank->max not min? That is the usual way to write a cyclic allocator over this kind of API. See the logic in __xa_alloc_cyclic()

Jason

author
#2 lore raw
Quoted 14 lines · Jason Gunthorpe

drivers/infiniband/hw/hns/hns_roce_qp.c static void hns_roce_ib_qp_event(struct hns_roce_qp *hr_qp, · line 183

Quoted 9 lines · Weihang Li
drivers/infiniband/hw/hns/hns_roce_qp.c +9 -0
+static int alloc_qpn_with_bankid(struct hns_roce_bank *bank, u8 bankid,
+				 unsigned long *qpn)
+{
+	int id;
+
+	id = ida_alloc_range(&bank->ida, bank->min, bank->max, GFP_KERNEL);
+	if (id < 0) {
+		id = ida_alloc_range(&bank->ida, bank->rsv_bot, bank->min,
+				     GFP_KERNEL);

Shouldn't this one be bank->max not min? That is the usual way to write a cyclic allocator over this kind of API. See the logic in __xa_alloc_cyclic()

Quoted 2 lines

Jason

Thanks for your advice, it's clearer to use bank->max here. We will refer to how __xa_alloc_cyclic() is implemented.

Weihang