Skip to contentSkip to conversation

qed: Enable RDMA relaxed ordering

PATCH In review Shai Malin 4 participants 9 messages Aug 22, 2021 – Aug 25, 2021

Subsystems (4)
lore
Share & tools
lore

Shai Malin

author
#0 lore raw

PATCH 1/1 qed: Enable RDMA relaxed ordering

Enable the RoCE and iWARP FW relaxed ordering.

---
 drivers/net/ethernet/qlogic/qed/qed_rdma.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/net/ethernet/qlogic/qed/qed_rdma.c +2 -0
index 4f4b79250a2b..496092655f26 100644
@@ -643,6 +643,8 @@ static int qed_rdma_start_fw(struct qed_hwfn *p_hwfn,
 				    cnq_id);
 	}
 
+	p_params_header->relaxed_ordering = 1;
+
 	return qed_spq_post(p_hwfn, p_ent, NULL);
 }
 
Signature · 1 line
2.22.0
#1 lore raw

+RDMA

Jakub, David

Can we please ask that everything directly or indirectly related to RDMA will be sent to linux-rdma@ too?

Quoted 13 lines · Shai Malin

Enable the RoCE and iWARP FW relaxed ordering.

---
 drivers/net/ethernet/qlogic/qed/qed_rdma.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/net/ethernet/qlogic/qed/qed_rdma.c +0 -0
index 4f4b79250a2b..496092655f26 100644

drivers/net/ethernet/qlogic/qed/qed_rdma.c static int qed_rdma_start_fw(struct qed_hwfn *p_hwfn, · line 646

Quoted 4 lines
drivers/net/ethernet/qlogic/qed/qed_rdma.c +1 -0
@@ -643,6 +643,8 @@ static int qed_rdma_start_fw(struct qed_hwfn *p_hwfn,
 				    cnq_id);
 	}
 
+	p_params_header->relaxed_ordering = 1;

Maybe it is only description that needs to be updated, but I would expect to see call to pcie_relaxed_ordering_enabled() before setting relaxed_ordering to always true.

If we are talking about RDMA, the IB_ACCESS_RELAXED_ORDERING flag should be taken into account too.

Thanks

Quoted 7 lines
drivers/net/ethernet/qlogic/qed/qed_rdma.c +1 -0
+
 	return qed_spq_post(p_hwfn, p_ent, NULL);
 }
 
Signature · 1 line
2.22.0
#2 lore raw
Quoted 22 lines · Leon Romanovsky

+RDMA

Jakub, David

Can we please ask that everything directly or indirectly related to RDMA will be sent to linux-rdma@ too?

Quoted 14 lines · Shai Malin

Enable the RoCE and iWARP FW relaxed ordering.

 drivers/net/ethernet/qlogic/qed/qed_rdma.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/net/ethernet/qlogic/qed/qed_rdma.c +1 -0
index 4f4b79250a2b..496092655f26 100644
+++ b/drivers/net/ethernet/qlogic/qed/qed_rdma.c
@@ -643,6 +643,8 @@ static int qed_rdma_start_fw(struct qed_hwfn *p_hwfn,
 				    cnq_id);
 	}
 
Quoted 8 lines

drivers/net/ethernet/qlogic/qed/qed_rdma.c static int qed_rdma_start_fw(struct qed_hwfn *p_hwfn, · line 646

Quoted 1 line
drivers/net/ethernet/qlogic/qed/qed_rdma.c +1 -0
+	p_params_header->relaxed_ordering = 1;

Maybe it is only description that needs to be updated, but I would expect to see call to pcie_relaxed_ordering_enabled() before setting relaxed_ordering to always true.

If we are talking about RDMA, the IB_ACCESS_RELAXED_ORDERING flag should be taken into account too.

Why does this file even exist in netdev? This whole struct qed_rdma_ops mess looks like another mis-design to support out of tree modules??

Jason

#3 lore raw
Quoted 44 lines

-----Original Message-----
From: Jason Gunthorpe <[email protected]>
Sent: Monday, August 23, 2021 4:34 PM
To: Leon Romanovsky <[email protected]>
Cc: Shai Malin <[email protected]>; [email protected]; [email protected]; [email protected]; Ariel Elior <[email protected]>; [email protected]; RDMA mailing list <linux- [email protected]>
Subject: [EXT] Re: [PATCH] qed: Enable RDMA relaxed ordering

External Email

----------------------------------------------------------------------

Quoted 15 lines · Leon Romanovsky

+RDMA

Jakub, David

Can we please ask that everything directly or indirectly related to RDMA will be sent to linux-rdma@ too?

Quoted 7 lines · Shai Malin

Enable the RoCE and iWARP FW relaxed ordering.

drivers/net/ethernet/qlogic/qed/qed_rdma.c | 2 ++

 1 file changed, 2 insertions(+)
Quoted 5 lines

drivers/net/ethernet/qlogic/qed/qed_rdma.c static int qed_rdma_start_fw(struct qed_hwfn

Quoted 5 lines
drivers/net/ethernet/qlogic/qed/qed_rdma.c +1 -0
index 4f4b79250a2b..496092655f26 100644
+++ b/drivers/net/ethernet/qlogic/qed/qed_rdma.c
@@ -643,6 +643,8 @@ static int qed_rdma_start_fw(struct qed_hwfn

*p_hwfn,

Quoted 9 lines

drivers/net/ethernet/qlogic/qed/qed_rdma.c static int qed_rdma_start_fw(struct qed_hwfn *p_hwfn, · line 646

Quoted 4 lines
drivers/net/ethernet/qlogic/qed/qed_rdma.c +1 -0
 				    cnq_id);
 	}
 
+	p_params_header->relaxed_ordering = 1;

Maybe it is only description that needs to be updated, but I would expect to see call to pcie_relaxed_ordering_enabled() before setting relaxed_ordering to always true.

Quoted 7 lines
Quoted 2 lines

If we are talking about RDMA, the IB_ACCESS_RELAXED_ORDERING flag should be taken into account too.

Why does this file even exist in netdev? This whole struct qed_rdma_ops mess looks like another mis-design to support out of tree modules??

Jason

Hi Jason,
qed_rdma_ops is not related to in tree / out of tree drivers. The qed is the core module which is used by the protocol drivers which drive this type of nic:
qede, qedr, qedi and qedf for ethernet, rdma, iscsi and fcoe respectively. qed mostly serves as a HW abstraction layer, hiding away the details of FW interaction and device register usage, and may also hold Linux specific things which are protocol agnostic, such as dcbx, sriov, debug data collection logic, etc. qed interacts with the protocol drivers through ops structs for many purposes (dcbx, ptp, sriov, etc). And also for rdma. It's just a way for us to separate the modules in a clean way.
Thanks,
Ariel

#4 lore raw
Quoted 50 lines · Ariel Elior
Quoted 42 lines

From: Jason Gunthorpe <[email protected]>
Sent: Monday, August 23, 2021 4:34 PM
To: Leon Romanovsky <[email protected]>
Cc: Shai Malin <[email protected]>; [email protected]; [email protected]; [email protected]; Ariel Elior <[email protected]>; [email protected]; RDMA mailing list <linux- [email protected]>
Subject: [EXT] Re: [PATCH] qed: Enable RDMA relaxed ordering

External Email

Quoted 15 lines · Leon Romanovsky

+RDMA

Jakub, David

Can we please ask that everything directly or indirectly related to RDMA will be sent to linux-rdma@ too?

Quoted 7 lines · Shai Malin

Enable the RoCE and iWARP FW relaxed ordering.

drivers/net/ethernet/qlogic/qed/qed_rdma.c | 2 ++

 1 file changed, 2 insertions(+)
Quoted 5 lines

drivers/net/ethernet/qlogic/qed/qed_rdma.c static int qed_rdma_start_fw(struct qed_hwfn

Quoted 5 lines
drivers/net/ethernet/qlogic/qed/qed_rdma.c +1 -0
index 4f4b79250a2b..496092655f26 100644
+++ b/drivers/net/ethernet/qlogic/qed/qed_rdma.c
@@ -643,6 +643,8 @@ static int qed_rdma_start_fw(struct qed_hwfn

*p_hwfn,

Quoted 9 lines

drivers/net/ethernet/qlogic/qed/qed_rdma.c static int qed_rdma_start_fw(struct qed_hwfn *p_hwfn, · line 646

Quoted 4 lines
drivers/net/ethernet/qlogic/qed/qed_rdma.c +1 -0
 				    cnq_id);
 	}
 
+	p_params_header->relaxed_ordering = 1;

Maybe it is only description that needs to be updated, but I would expect to see call to pcie_relaxed_ordering_enabled() before setting relaxed_ordering to always true.

Quoted 7 lines
Quoted 2 lines

If we are talking about RDMA, the IB_ACCESS_RELAXED_ORDERING flag should be taken into account too.

Why does this file even exist in netdev? This whole struct qed_rdma_ops mess looks like another mis-design to support out of tree modules??

Jason

Quoted 10 lines

Hi Jason,
qed_rdma_ops is not related to in tree / out of tree drivers. The qed is the core module which is used by the protocol drivers which drive this type of nic:
qede, qedr, qedi and qedf for ethernet, rdma, iscsi and fcoe respectively. qed mostly serves as a HW abstraction layer, hiding away the details of FW interaction and device register usage, and may also hold Linux specific things which are protocol agnostic, such as dcbx, sriov, debug data collection logic, etc. qed interacts with the protocol drivers through ops structs for many purposes (dcbx, ptp, sriov, etc). And also for rdma. It's just a way for us to separate the modules in a clean way.

Delete the ops struct.

Move the RDMA functions to the RDMA module

Directly export the core functions needed to make that work

Two halfs of the same dirver do not and should not have an ops structure ABI between them.

Jason

#5 lore raw
Quoted 62 lines · Jason Gunthorpe
Quoted 50 lines · Ariel Elior
Quoted 42 lines

From: Jason Gunthorpe <[email protected]>
Sent: Monday, August 23, 2021 4:34 PM
To: Leon Romanovsky <[email protected]>
Cc: Shai Malin <[email protected]>; [email protected]; [email protected]; [email protected]; Ariel Elior <[email protected]>; [email protected]; RDMA mailing list <linux- [email protected]>
Subject: [EXT] Re: [PATCH] qed: Enable RDMA relaxed ordering

External Email

Quoted 15 lines · Leon Romanovsky

+RDMA

Jakub, David

Can we please ask that everything directly or indirectly related to RDMA will be sent to linux-rdma@ too?

Quoted 7 lines · Shai Malin

Enable the RoCE and iWARP FW relaxed ordering.

drivers/net/ethernet/qlogic/qed/qed_rdma.c | 2 ++

 1 file changed, 2 insertions(+)
Quoted 5 lines

drivers/net/ethernet/qlogic/qed/qed_rdma.c static int qed_rdma_start_fw(struct qed_hwfn

Quoted 5 lines
drivers/net/ethernet/qlogic/qed/qed_rdma.c +1 -0
index 4f4b79250a2b..496092655f26 100644
+++ b/drivers/net/ethernet/qlogic/qed/qed_rdma.c
@@ -643,6 +643,8 @@ static int qed_rdma_start_fw(struct qed_hwfn

*p_hwfn,

Quoted 9 lines

drivers/net/ethernet/qlogic/qed/qed_rdma.c static int qed_rdma_start_fw(struct qed_hwfn *p_hwfn, · line 646

Quoted 4 lines
drivers/net/ethernet/qlogic/qed/qed_rdma.c +1 -0
 				    cnq_id);
 	}
 
+	p_params_header->relaxed_ordering = 1;

Maybe it is only description that needs to be updated, but I would expect to see call to pcie_relaxed_ordering_enabled() before setting relaxed_ordering to always true.

Quoted 7 lines
Quoted 2 lines

If we are talking about RDMA, the IB_ACCESS_RELAXED_ORDERING flag should be taken into account too.

Why does this file even exist in netdev? This whole struct qed_rdma_ops mess looks like another mis-design to support out of tree modules??

Jason

Quoted 10 lines

Hi Jason,
qed_rdma_ops is not related to in tree / out of tree drivers. The qed is the core module which is used by the protocol drivers which drive this type of nic:
qede, qedr, qedi and qedf for ethernet, rdma, iscsi and fcoe respectively. qed mostly serves as a HW abstraction layer, hiding away the details of FW interaction and device register usage, and may also hold Linux specific things which are protocol agnostic, such as dcbx, sriov, debug data collection logic, etc. qed interacts with the protocol drivers through ops structs for many purposes (dcbx, ptp, sriov, etc). And also for rdma. It's just a way for us to separate the modules in a clean way.

Quoted 8 lines

Delete the ops struct.

Move the RDMA functions to the RDMA module

Directly export the core functions needed to make that work

Two halfs of the same dirver do not and should not have an ops structure ABI between them.

Yea, I read drivers/net/ethernet/qlogic/qed/qed_rdma.c and have hard time to believe that hiding RDMA objects and code from the RDMA community can be counted as "a clean way".

Thanks

Quoted 2 lines

Jason

#6 lore raw
Quoted 82 lines

-----Original Message-----
From: Leon Romanovsky <[email protected]>
Sent: Tuesday, August 24, 2021 3:25 PM
To: Jason Gunthorpe <[email protected]>
Cc: Ariel Elior <[email protected]>; Shai Malin <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; RDMA mailing list <[email protected]>
Subject: Re: [EXT] Re: [PATCH] qed: Enable RDMA relaxed ordering

Quoted 44 lines · Jason Gunthorpe
Quoted 43 lines · Ariel Elior
Quoted 43 lines

From: Jason Gunthorpe <[email protected]>
Sent: Monday, August 23, 2021 4:34 PM
To: Leon Romanovsky <[email protected]>
Cc: Shai Malin <[email protected]>; [email protected]; [email protected]; [email protected]; Ariel Elior <[email protected]>; [email protected]; RDMA mailing list <linux- [email protected]>
Subject: [EXT] Re: [PATCH] qed: Enable RDMA relaxed ordering

External Email

Quoted 15 lines · Leon Romanovsky

+RDMA

Jakub, David

Can we please ask that everything directly or indirectly related to RDMA will be sent to linux-rdma@ too?

Quoted 7 lines · Shai Malin

Enable the RoCE and iWARP FW relaxed ordering.

drivers/net/ethernet/qlogic/qed/qed_rdma.c | 2 ++

 1 file changed, 2 insertions(+)
Quoted 6 lines
Quoted 6 lines
drivers/net/ethernet/qlogic/qed/qed_rdma.c +1 -0
index 4f4b79250a2b..496092655f26 100644
+++ b/drivers/net/ethernet/qlogic/qed/qed_rdma.c
@@ -643,6 +643,8 @@ static int qed_rdma_start_fw(struct

qed_hwfn

*p_hwfn,

Quoted 9 lines

drivers/net/ethernet/qlogic/qed/qed_rdma.c static int qed_rdma_start_fw(struct qed_hwfn *p_hwfn, · line 646

Quoted 4 lines
drivers/net/ethernet/qlogic/qed/qed_rdma.c +1 -0
 				    cnq_id);
 	}
 
+	p_params_header->relaxed_ordering = 1;

Maybe it is only description that needs to be updated, but I would expect to see call to pcie_relaxed_ordering_enabled() before setting relaxed_ordering to always true.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines

If we are talking about RDMA, the IB_ACCESS_RELAXED_ORDERING flag should be taken into account too.

Why does this file even exist in netdev? This whole struct qed_rdma_ops mess looks like another mis-design to support out of

tree modules??

Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

Jason

Hi Jason,
qed_rdma_ops is not related to in tree / out of tree drivers. The qed is the core module which is used by the protocol drivers which drive

this type of nic:

Quoted 7 lines
Quoted 7 lines

qede, qedr, qedi and qedf for ethernet, rdma, iscsi and fcoe respectively. qed mostly serves as a HW abstraction layer, hiding away the details of FW interaction and device register usage, and may also hold Linux specific things which are protocol agnostic, such as dcbx, sriov, debug data collection logic, etc. qed interacts with the protocol drivers through ops structs for many purposes (dcbx, ptp, sriov, etc). And also for rdma. It's just a way for us to separate the modules in a

clean way.

Quoted 5 lines

Delete the ops struct.

Move the RDMA functions to the RDMA module

Quoted 8 lines
Quoted 4 lines

Directly export the core functions needed to make that work

Two halfs of the same dirver do not and should not have an ops structure ABI between them.

Yea, I read drivers/net/ethernet/qlogic/qed/qed_rdma.c and have hard time to believe that hiding RDMA objects and code from the RDMA community can be counted as "a clean way".

Hi Jason, Leon

I certainly see your point, and understand the motivation to have rdma content in the rdma tree. We will start work on refactoring the (day 1) driver design to have more rdma logic in the rdma driver and invoke the core module when needed. Changing ops to exported functions will also be part of that.

But realistically I don't think we can move it all. Please understand that the core module has many responsibilities which must take RDMA components into account, but are not just rdma specific (the same logic is applied for the other protocol drivers).

A few examples for this might be laying out host memory for connection contexts, computing bar offsets, computing resource amounts and allocating resources for VFs/PFs, etc.

I think we can definitely move some of the RDMA logic from core module to the rdma driver (as in the case of this patchset) and I understand it makes more sense that way. But I can think of multiple code areas where this would be very difficult.

The current design is for the core module to own data structures and logic for device configuration and manipulation. These flows/data-structures are triggered/populated from multiple entry points: some at the low level part of protocol flows (e.g. create QP) which can easily transition to be an exported function as you directed, but other entry points may also be activated earlier e.g. when device is initialized, even before rdma driver is loaded (based on device configuration information, for example) in which case we would not be able to invoke functionality residing in the rdma driver, but still have to populate data structures, invoke FW flows, configure registers, etc. which have to do with RDMA.

Additionally, the qedr RDMA driver services both roce and iwarp, which means there are TCP related flows/data structures which are shared between it and our iSCSI driver qedi. This is nicely handled in the common module avoiding any code duplication. Ripping it out and locating it in the protocol driver would be difficult to perform and hard to maintain across the two trees.

Likewise functionality like light l2 queues, dcbx, sriov are shared amongst all the protocol drivers. Exposing the functionality through export instead of ops is no problem, but moving the logic outside of the core module to a specific protocol is both a considerable design change and may lead to code duplication or some very convoluted flows.

In our view the qed/qede/qedr/qedi/qedf are separate drivers, hence we used function pointer structures for the communication between them. We use hierarchies of structures of function pointers to group toghether those which have common purposes (dcbx, ll2, Ethernet, rdma). Changing that to flat exported functions for the RDMA protocol is no problem if it is preferred by you.

In summary - we got the message and will work on it, but this is no small task and may take some time, and will likely not result in total removal of any mention whatsoever of rdma from the core module (but will reduce it considerably).

Quoted 5 lines

Thanks

Quoted 2 lines

Jason

#7 lore raw
Quoted 5 lines · Ariel Elior

In our view the qed/qede/qedr/qedi/qedf are separate drivers, hence we used function pointer structures for the communication between them. We use hierarchies of structures of function pointers to group toghether those which have common purposes (dcbx, ll2, Ethernet, rdma). Changing that to flat exported functions for the RDMA protocol is no problem if it is preferred by you.

I wouldn't twist the driver into knots, but you definately should not be using function pointers when there is only one implementation, eliminating that would be a fine start and looks straightforward.

Many of the functions in the rdma ops do not look complicated to move, yes, it moves around the layering a bit, but that is OK and probably more maintainable in the end. eg modify_qp seems fairly disconnected at the first couple layers of function calls.

Quoted 4 lines

In summary - we got the message and will work on it, but this is no small task and may take some time, and will likely not result in total removal of any mention whatsoever of rdma from the core module (but will reduce it considerably).

I wouldn't go for complete removal, you just need to have a core driver with an exported API that makes some sense for the device.

eg looking at a random op

qed_iwarp_set_engine_affin()

Is an "rdma" function but all it does is call
qed_llh_set_ppfid_affinity()

So export qed_llh and move the qed_iwarp to the rdma driver

etc

Jason

#8 lore raw
Quoted 40 lines

-----Original Message-----
From: Jason Gunthorpe <[email protected]>
Sent: Tuesday, August 24, 2021 10:42 PM
To: Ariel Elior <[email protected]>
Cc: Leon Romanovsky <[email protected]>; Shai Malin <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; RDMA mailing list <linux- [email protected]>
Subject: Re: [EXT] Re: [PATCH] qed: Enable RDMA relaxed ordering

Quoted 5 lines · Ariel Elior

In our view the qed/qede/qedr/qedi/qedf are separate drivers, hence we used function pointer structures for the communication between them. We use hierarchies of structures of function pointers to group toghether those which have common purposes (dcbx, ll2, Ethernet, rdma). Changing that to flat exported functions for the RDMA protocol is no

problem if it is preferred by you.

I wouldn't twist the driver into knots, but you definately should not be using function pointers when there is only one implementation, eliminating that would be a fine start and looks straightforward.

Many of the functions in the rdma ops do not look complicated to move, yes, it moves around the layering a bit, but that is OK and probably more maintainable in the end. eg modify_qp seems fairly disconnected at the first couple layers of function calls.

Quoted 4 lines

In summary - we got the message and will work on it, but this is no small task and may take some time, and will likely not result in total removal of any mention whatsoever of rdma from the core module (but will reduce it considerably).

I wouldn't go for complete removal, you just need to have a core driver with an exported API that makes some sense for the device.

eg looking at a random op

Quoted 8 lines

qed_iwarp_set_engine_affin()

Is an "rdma" function but all it does is call
qed_llh_set_ppfid_affinity()

So export qed_llh and move the qed_iwarp to the rdma driver

etc

Got it, and makes sense to me. I get the point on single instance of function pointers being redundant. We will start work on the necessary redesign right away. Meanwhile you may see a few more critical fixes/small features coming from us which are already queued up internally on our end, which I hope can be accepted before we perform the changes discussed here.
Thanks,
Ariel

Quoted 2 lines

Jason