Skip to contentSkip to conversation

Misc changes for siw

PATCH In review Guoqing Jiang 2 participants 11 messages Nov 29, 2023 – Dec 3, 2023

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover Misc changes for siw

Hi,

The first two patches try to reduce memory usage, the third one fix test_query_rc_qp failure, and the last one clean up code a bit.

Please review.

Thanks,
Guoqing

Guoqing Jiang (4):
RDMA/siw: Move tx_cpu ahead
RDMA/siw: Reduce memory usage of struct siw_rx_stream RDMA/siw: Set qp_state in siw_query_qp RDMA/siw: Call orq_get_current if possible

 drivers/infiniband/sw/siw/siw.h       | 10 +++++-----
 drivers/infiniband/sw/siw/siw_verbs.c | 10 ++++++++++
 2 files changed, 15 insertions(+), 5 deletions(-)

base-commit: 50af5d12f7e24b85fc10270d7700f4aa1b20b8e4

Signature · 1 line
2.35.3
#1 lore raw

PATCH 1/4 RDMA/siw: Move tx_cpu ahead

We can reduce one cacheline for the usage of struct siw_qp.

Before,

	/* size: 1928, cachelines: 31, members: 38 */
	/* sum members: 1920, holes: 2, sum holes: 8 */
	/* paddings: 4, sum paddings: 13 */
	/* forced alignments: 3 */

after

	/* size: 1920, cachelines: 30, members: 38 */
	/* paddings: 4, sum paddings: 13 */
	/* forced alignments: 3 */
---
 drivers/infiniband/sw/siw/siw.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/infiniband/sw/siw/siw.h +1 -1
index b36d1ec25327..d14bb965af75 100644
@@ -417,10 +417,10 @@ struct siw_iwarp_tx {
 struct siw_qp {
 	struct ib_qp base_qp;
 	struct siw_device *sdev;
+	int tx_cpu;
 	struct kref ref;
 	struct completion qp_free;
 	struct list_head devq;
-	int tx_cpu;
 	struct siw_qp_attrs attrs;
 
 	struct siw_cep *cep;
Signature · 1 line
2.35.3
Acked-by on v1 1/4
#2 lore raw

RE: [PATCH 1/4] RDMA/siw: Move tx_cpu ahead

Quoted 47 lines

-----Original Message-----
From: Guoqing Jiang <[email protected]>
Sent: Wednesday, November 29, 2023 4:24 AM
To: Bernard Metzler <[email protected]>; [email protected]; [email protected]
Cc: [email protected]; [email protected]
Subject: [EXTERNAL] [PATCH 1/4] RDMA/siw: Move tx_cpu ahead

We can reduce one cacheline for the usage of struct siw_qp.

Before,

	/* size: 1928, cachelines: 31, members: 38 */
	/* sum members: 1920, holes: 2, sum holes: 8 */
	/* paddings: 4, sum paddings: 13 */
	/* forced alignments: 3 */

after

	/* size: 1920, cachelines: 30, members: 38 */
	/* paddings: 4, sum paddings: 13 */
	/* forced alignments: 3 */
---
 drivers/infiniband/sw/siw/siw.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/infiniband/sw/siw/siw.h +1 -2
index b36d1ec25327..d14bb965af75 100644
@@ -417,10 +417,10 @@ struct siw_iwarp_tx {
 struct siw_qp {
 	struct ib_qp base_qp;
 	struct siw_device *sdev;
+	int tx_cpu;
 	struct kref ref;
 	struct completion qp_free;
 	struct list_head devq;
-	int tx_cpu;
 	struct siw_qp_attrs attrs;
 
 	struct siw_cep *cep;
--
Quoted 1 line

2.35.3

Looks good.

#3 lore raw

PATCH 2/4 RDMA/siw: Reduce memory usage of struct siw_rx_stream

In reply to Guoqing Jiang

We can reduce the memory of the struct by move some of it's member.

Before,

	/* size: 144, cachelines: 3, members: 17 */
	/* sum members: 124, holes: 3, sum holes: 12 */
	/* sum bitfield members: 7 bits (0 bytes) */
	/* padding: 7 */
	/* bit_padding: 1 bits */

After

	/* size: 128, cachelines: 2, members: 17 */
	/* padding: 3 */
	/* bit_padding: 1 bits */
---
 drivers/infiniband/sw/siw/siw.h | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
drivers/infiniband/sw/siw/siw.h +3 -3
index d14bb965af75..2edba2a864bb 100644
@@ -288,10 +288,11 @@ struct siw_rx_stream {
 	int skb_offset; /* offset in skb */
 	int skb_copied; /* processed bytes in skb */
 
+	enum siw_rx_state state;
+
 	union iwarp_hdr hdr;
 	struct mpa_trailer trailer;
-
-	enum siw_rx_state state;
+	struct shash_desc *mpa_crc_hd;
 
 	/*
 	 * For each FPDU, main RX loop runs through 3 stages:
@@ -313,7 +314,6 @@ struct siw_rx_stream {
 	u64 ddp_to;
 	u32 inval_stag; /* Stag to be invalidated */
 
-	struct shash_desc *mpa_crc_hd;
 	u8 rx_suspend : 1;
 	u8 pad : 2; /* # of pad bytes expected */
 	u8 rdmap_op : 4; /* opcode of current frame */
Signature · 1 line
2.35.3
Acked-by on v1 2/4
#4 lore raw

RE: [PATCH 2/4] RDMA/siw: Reduce memory usage of struct siw_rx_stream

Quoted 60 lines

-----Original Message-----
From: Guoqing Jiang <[email protected]>
Sent: Wednesday, November 29, 2023 4:24 AM
To: Bernard Metzler <[email protected]>; [email protected]; [email protected]
Cc: [email protected]; [email protected]
Subject: [EXTERNAL] [PATCH 2/4] RDMA/siw: Reduce memory usage of struct siw_rx_stream

We can reduce the memory of the struct by move some of it's member.

Before,

	/* size: 144, cachelines: 3, members: 17 */
	/* sum members: 124, holes: 3, sum holes: 12 */
	/* sum bitfield members: 7 bits (0 bytes) */
	/* padding: 7 */
	/* bit_padding: 1 bits */

After

	/* size: 128, cachelines: 2, members: 17 */
	/* padding: 3 */
	/* bit_padding: 1 bits */
---
 drivers/infiniband/sw/siw/siw.h | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
drivers/infiniband/sw/siw/siw.h +3 -4
index d14bb965af75..2edba2a864bb 100644
@@ -288,10 +288,11 @@ struct siw_rx_stream {
 	int skb_offset; /* offset in skb */
 	int skb_copied; /* processed bytes in skb */
 
+	enum siw_rx_state state;
+
 	union iwarp_hdr hdr;
 	struct mpa_trailer trailer;
-
-	enum siw_rx_state state;
+	struct shash_desc *mpa_crc_hd;
 
 	/*
 	 * For each FPDU, main RX loop runs through 3 stages:
@@ -313,7 +314,6 @@ struct siw_rx_stream {
 	u64 ddp_to;
 	u32 inval_stag; /* Stag to be invalidated */
 
-	struct shash_desc *mpa_crc_hd;
 	u8 rx_suspend : 1;
 	u8 pad : 2; /* # of pad bytes expected */
 	u8 rdmap_op : 4; /* opcode of current frame */
--
Quoted 1 line

2.35.3

Looks good.

#5 lore raw

PATCH 3/4 RDMA/siw: Set qp_state in siw_query_qp

In reply to Guoqing Jiang

Run test_query_rc_qp against siw failed since siw didn't set qp_state accordingly. To address it, introduce siw_qp_state_to_ib_qp_state which convert SIW_QP_STATE_IDLE to IB_QPS_INIT which is similar as in cxgb4.

rdma-core# ./build/bin/run_tests.py --dev siw0 tests.test_qp.QPTest.test_query_rc_qp -v test_query_rc_qp (tests.test_qp.QPTest)
Queries an RC QP after creation. Verifies that its properties are as ... FAIL

======================================================================
FAIL: test_query_rc_qp (tests.test_qp.QPTest)
Queries an RC QP after creation. Verifies that its properties are as
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/gjiang/rdma-core/tests/test_qp.py", line 284, in test_query_rc_qp
self.query_qp_common_test(e.IBV_QPT_RC)
File "/home/gjiang/rdma-core/tests/test_qp.py", line 265, in query_qp_common_test
self.verify_qp_attrs(caps, e.IBV_QPS_INIT, qp_init_attr, qp_attr)
File "/home/gjiang/rdma-core/tests/test_qp.py", line 239, in verify_qp_attrs
self.assertEqual(state, attr.qp_state)
AssertionError: <ibv_qp_state.IBV_QPS_INIT: 1> != 0

----------------------------------------------------------------------
Ran 1 test in 0.057s

FAILED (failures=1)

---
 drivers/infiniband/sw/siw/siw_verbs.c | 10 ++++++++++
 1 file changed, 10 insertions(+)
drivers/infiniband/sw/siw/siw_verbs.c +10 -0
index dca6a155523d..233985434cfd 100644
@@ -19,6 +19,15 @@
 #include "siw_verbs.h"
 #include "siw_mem.h"
 
+static int siw_qp_state_to_ib_qp_state[IB_QPS_ERR] = {
+	[SIW_QP_STATE_IDLE] = IB_QPS_INIT,
+	[SIW_QP_STATE_RTR] = IB_QPS_RTR,
+	[SIW_QP_STATE_RTS] = IB_QPS_RTS,
+	[SIW_QP_STATE_CLOSING] = IB_QPS_SQD,
+	[SIW_QP_STATE_TERMINATE] = IB_QPS_SQE,
+	[SIW_QP_STATE_ERROR] = IB_QPS_ERR
+};
+
 static int ib_qp_state_to_siw_qp_state[IB_QPS_ERR + 1] = {
 	[IB_QPS_RESET] = SIW_QP_STATE_IDLE,
 	[IB_QPS_INIT] = SIW_QP_STATE_IDLE,
@@ -504,6 +513,7 @@ int siw_query_qp(struct ib_qp *base_qp, struct ib_qp_attr *qp_attr,
 	} else {
 		return -EINVAL;
 	}
+	qp_attr->qp_state = siw_qp_state_to_ib_qp_state[qp->attrs.state];
 	qp_attr->cap.max_inline_data = SIW_MAX_INLINE;
 	qp_attr->cap.max_send_wr = qp->attrs.sq_size;
 	qp_attr->cap.max_send_sge = qp->attrs.sq_max_sges;
Signature · 1 line
2.35.3
#6 lore raw

RE: [PATCH 3/4] RDMA/siw: Set qp_state in siw_query_qp

Quoted 36 lines

-----Original Message-----
From: Guoqing Jiang <[email protected]>
Sent: Wednesday, November 29, 2023 4:24 AM
To: Bernard Metzler <[email protected]>; [email protected]; [email protected]
Cc: [email protected]; [email protected]
Subject: [EXTERNAL] [PATCH 3/4] RDMA/siw: Set qp_state in siw_query_qp

Run test_query_rc_qp against siw failed since siw didn't set qp_state accordingly. To address it, introduce siw_qp_state_to_ib_qp_state which convert SIW_QP_STATE_IDLE to IB_QPS_INIT which is similar as in cxgb4.

rdma-core# ./build/bin/run_tests.py --dev siw0 tests.test_qp.QPTest.test_query_rc_qp -v
test_query_rc_qp (tests.test_qp.QPTest)
Queries an RC QP after creation. Verifies that its properties are as ... FAIL

======================================================================
FAIL: test_query_rc_qp (tests.test_qp.QPTest)
Queries an RC QP after creation. Verifies that its properties are as
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/gjiang/rdma-core/tests/test_qp.py", line 284, in
test_query_rc_qp
self.query_qp_common_test(e.IBV_QPT_RC)
File "/home/gjiang/rdma-core/tests/test_qp.py", line 265, in
query_qp_common_test
self.verify_qp_attrs(caps, e.IBV_QPS_INIT, qp_init_attr, qp_attr)
File "/home/gjiang/rdma-core/tests/test_qp.py", line 239, in
verify_qp_attrs
self.assertEqual(state, attr.qp_state)
AssertionError: <ibv_qp_state.IBV_QPS_INIT: 1> != 0

Quoted 6 lines

----------------------------------------------------------------------
Ran 1 test in 0.057s

FAILED (failures=1)

Very nice finding! How could that remain undetected. Probably no real application checks QP state 😉

Quoted 10 lines
---
 drivers/infiniband/sw/siw/siw_verbs.c | 10 ++++++++++
 1 file changed, 10 insertions(+)
drivers/infiniband/sw/siw/siw_verbs.c +0 -0
index dca6a155523d..233985434cfd 100644

drivers/infiniband/sw/siw/siw_verbs.c line 22

Quoted 4 lines
drivers/infiniband/sw/siw/siw_verbs.c +1 -0
@@ -19,6 +19,15 @@
 #include "siw_verbs.h"
 #include "siw_mem.h"
 
+static int siw_qp_state_to_ib_qp_state[IB_QPS_ERR] = {

You may better use SIW_QP_STATE_COUNT for the size of the array

Quoted 22 lines
drivers/infiniband/sw/siw/siw_verbs.c +9 -0
+	[SIW_QP_STATE_IDLE] = IB_QPS_INIT,
+	[SIW_QP_STATE_RTR] = IB_QPS_RTR,
+	[SIW_QP_STATE_RTS] = IB_QPS_RTS,
+	[SIW_QP_STATE_CLOSING] = IB_QPS_SQD,
+	[SIW_QP_STATE_TERMINATE] = IB_QPS_SQE,
+	[SIW_QP_STATE_ERROR] = IB_QPS_ERR
+};
+
 static int ib_qp_state_to_siw_qp_state[IB_QPS_ERR + 1] = {
 	[IB_QPS_RESET] = SIW_QP_STATE_IDLE,
 	[IB_QPS_INIT] = SIW_QP_STATE_IDLE,
@@ -504,6 +513,7 @@ int siw_query_qp(struct ib_qp *base_qp, struct
 ib_qp_attr *qp_attr,
 	} else {
 		return -EINVAL;
 	}
+	qp_attr->qp_state = siw_qp_state_to_ib_qp_state[qp->attrs.state];
 	qp_attr->cap.max_inline_data = SIW_MAX_INLINE;
 	qp_attr->cap.max_send_wr = qp->attrs.sq_size;

qp_attr->cap.max_send_sge = qp->attrs.sq_max_sges;
--
2.35.3

#7 lore raw

Re: [PATCH 3/4] RDMA/siw: Set qp_state in siw_query_qp

Hi Bernard,

Quoted 34 lines · Bernard Metzler
Quoted 34 lines

-----Original Message-----
From: Guoqing Jiang <[email protected]>
Sent: Wednesday, November 29, 2023 4:24 AM
To: Bernard Metzler <[email protected]>; [email protected]; [email protected]
Cc: [email protected]; [email protected]
Subject: [EXTERNAL] [PATCH 3/4] RDMA/siw: Set qp_state in siw_query_qp

Run test_query_rc_qp against siw failed since siw didn't set qp_state accordingly. To address it, introduce siw_qp_state_to_ib_qp_state which convert SIW_QP_STATE_IDLE to IB_QPS_INIT which is similar as in cxgb4.

rdma-core# ./build/bin/run_tests.py --dev siw0 tests.test_qp.QPTest.test_query_rc_qp -v
test_query_rc_qp (tests.test_qp.QPTest)
Queries an RC QP after creation. Verifies that its properties are as ... FAIL

======================================================================
FAIL: test_query_rc_qp (tests.test_qp.QPTest)
Queries an RC QP after creation. Verifies that its properties are as
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/gjiang/rdma-core/tests/test_qp.py", line 284, in
test_query_rc_qp
self.query_qp_common_test(e.IBV_QPT_RC)
File "/home/gjiang/rdma-core/tests/test_qp.py", line 265, in
query_qp_common_test
self.verify_qp_attrs(caps, e.IBV_QPS_INIT, qp_init_attr, qp_attr)
File "/home/gjiang/rdma-core/tests/test_qp.py", line 239, in
verify_qp_attrs
self.assertEqual(state, attr.qp_state)
AssertionError: <ibv_qp_state.IBV_QPS_INIT: 1> != 0

Quoted 8 lines
Quoted 6 lines

----------------------------------------------------------------------
Ran 1 test in 0.057s

FAILED (failures=1)

Very nice finding! How could that remain undetected. Probably no real application checks QP state 😉

My guess is no one run the test after the commit 😉.

698f2ae80476 tests: Fix comparing qp_state for iWARP providers.

Quoted 13 lines
Quoted 13 lines
---
  drivers/infiniband/sw/siw/siw_verbs.c | 10 ++++++++++
  1 file changed, 10 insertions(+)
drivers/infiniband/sw/siw/siw_verbs.c +0 -0
index dca6a155523d..233985434cfd 100644
@@ -19,6 +19,15 @@
  #include "siw_verbs.h"
  #include "siw_mem.h"
 
Quoted 3 lines

drivers/infiniband/sw/siw/siw_verbs.c line 22

Quoted 1 line
drivers/infiniband/sw/siw/siw_verbs.c +1 -0
+static int siw_qp_state_to_ib_qp_state[IB_QPS_ERR] = {

You may better use SIW_QP_STATE_COUNT for the size of the array

Right, fully agree, will change it in next version after you finish review this version.

Thanks,
Guoqing

#8 lore raw

PATCH 4/4 RDMA/siw: Call orq_get_current if possible

In reply to Guoqing Jiang

We can call it in siw_orq_empty.

---
 drivers/infiniband/sw/siw/siw.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/infiniband/sw/siw/siw.h +1 -1
index 2edba2a864bb..75253f2b3e3d 100644
@@ -657,7 +657,7 @@ static inline struct siw_sqe *orq_get_free(struct siw_qp *qp)
 
 static inline int siw_orq_empty(struct siw_qp *qp)
 {
-	return qp->orq[qp->orq_get % qp->attrs.orq_size].flags == 0 ? 1 : 0;
+	return orq_get_current(qp)->flags == 0 ? 1 : 0;
 }
 
 static inline struct siw_sqe *irq_alloc_free(struct siw_qp *qp)
Signature · 1 line
2.35.3
Acked-by on v1 4/4
#9 lore raw

RE: [PATCH 4/4] RDMA/siw: Call orq_get_current if possible

Quoted 30 lines

-----Original Message-----
From: Guoqing Jiang <[email protected]>
Sent: Wednesday, November 29, 2023 4:24 AM
To: Bernard Metzler <[email protected]>; [email protected]; [email protected]
Cc: [email protected]; [email protected]
Subject: [EXTERNAL] [PATCH 4/4] RDMA/siw: Call orq_get_current if possible

We can call it in siw_orq_empty.

---
 drivers/infiniband/sw/siw/siw.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/infiniband/sw/siw/siw.h +1 -1
index 2edba2a864bb..75253f2b3e3d 100644
@@ -657,7 +657,7 @@ static inline struct siw_sqe *orq_get_free(struct
 siw_qp *qp)
 
 static inline int siw_orq_empty(struct siw_qp *qp)
 {
-	return qp->orq[qp->orq_get % qp->attrs.orq_size].flags == 0 ? 1 : 0;
+	return orq_get_current(qp)->flags == 0 ? 1 : 0;
 }
 
Quoted 3 lines

static inline struct siw_sqe *irq_alloc_free(struct siw_qp *qp)
--
2.35.3

Please change the commit message. Something like
'Use orq_get_current() in siw_orq_empty()'.

Otherwise looks good!

#10 lore raw

Re: [PATCH 4/4] RDMA/siw: Call orq_get_current if possible

Quoted 29 lines · Bernard Metzler
Quoted 29 lines

-----Original Message-----
From: Guoqing Jiang <[email protected]>
Sent: Wednesday, November 29, 2023 4:24 AM
To: Bernard Metzler <[email protected]>; [email protected]; [email protected]
Cc: [email protected]; [email protected]
Subject: [EXTERNAL] [PATCH 4/4] RDMA/siw: Call orq_get_current if possible

We can call it in siw_orq_empty.

---
  drivers/infiniband/sw/siw/siw.h | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
drivers/infiniband/sw/siw/siw.h +1 -1
index 2edba2a864bb..75253f2b3e3d 100644
@@ -657,7 +657,7 @@ static inline struct siw_sqe *orq_get_free(struct
 siw_qp *qp)
 
  static inline int siw_orq_empty(struct siw_qp *qp)
  {
-	return qp->orq[qp->orq_get % qp->attrs.orq_size].flags == 0 ? 1 : 0;
+	return orq_get_current(qp)->flags == 0 ? 1 : 0;
  }
 
Quoted 5 lines
Quoted 3 lines

static inline struct siw_sqe *irq_alloc_free(struct siw_qp *qp)
--
2.35.3

Please change the commit message. Something like
'Use orq_get_current() in siw_orq_empty()'.

Ok, will do.

Quoted 3 lines

Otherwise looks good!

Thanks for your review!

Guoqing