RDMA/core: Fix corrupted SL on passive side
PATCH 1/1 RDMA/core: Fix corrupted SL on passive side
On RoCE systems, a CM REQ contains a Primary Hop Limit > 1 and Primary Subnet Local is zero.
In cm_req_handler(), the cm_process_routed_req() function is called. Since the Primary Subnet Local value is zero in the request, and since this is RoCE (Primary Local LID is permissive), the following statement will be executed:
IBA_SET(CM_REQ_PRIMARY_SL, req_msg, wc->sl);
This corrupts SL in req_msg if it was different from zero. In other words, a request to setup a connection using an SL != zero, will not be honored, and a connection using SL zero will be created instead.
Fixed by not calling cm_process_routed_req() on RoCE systems.
--- drivers/infiniband/core/cm.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/infiniband/core/cm.c +2 -1
@@ -2138,7 +2138,8 @@ static int cm_req_handler(struct cm_work *work) goto destroy; } -cm_process_routed_req (req_msg, work->mad_recv_wc->wc); +if (cm_id_priv->av.ah_attr.type !=RDMA_AH_ATTR_TYPE_ROCE ) +cm_process_routed_req (req_msg, work->mad_recv_wc->wc);memset (&work->path[0 ],0 ,sizeof (work->path[0 ]));if (cm_req_has_alt_path (req_msg))
Signature · 1 line
1.8.3.1
Jason Gunthorpe
maintainerQuoted 25 lines · Håkon Bugge
On RoCE systems, a CM REQ contains a Primary Hop Limit > 1 and Primary Subnet Local is zero.
In
cm_req_handler(), thecm_process_routed_req()function is called. Since the Primary Subnet Local value is zero in the request, and since this is RoCE (Primary Local LID is permissive), the following statement will be executed:IBA_SET(CM_REQ_PRIMARY_SL, req_msg, wc->sl);This corrupts SL in
req_msgif it was different from zero. In other words, a request to setup a connection using an SL != zero, will not be honored, and a connection using SL zero will be created instead.Fixed by not calling
cm_process_routed_req()on RoCE systems.Fixes 3971c9f6dbf2 ("IB/cm: Add interim support for routed paths")Signed-off-by Håkon Bugge <[email protected]>drivers/infiniband/core/cm.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)drivers/infiniband/core/cm.c +1
-0+++ b/drivers/infiniband/core/cm.c
drivers/infiniband/core/cm.c static int cm_req_handler(struct cm_work *work) · line 2142
Quoted 6 lines
drivers/infiniband/core/cm.c +2
-1@@ -2138,7 +2138,8 @@ static int cm_req_handler(struct cm_work *work) goto destroy; } -cm_process_routed_req (req_msg, work->mad_recv_wc->wc); +if (cm_id_priv->av.ah_attr.type !=RDMA_AH_ATTR_TYPE_ROCE ) +cm_process_routed_req (req_msg, work->mad_recv_wc->wc);
why use ah_attr.type when a few lines below we have:
if (gid_attr &&
rdma_protocol_roce(work->port->cm_dev->ib_device,
work->port->port_num)) {?
I suspect you can just move this into the else?
Jason
Quoted 39 lines · Jason Gunthorpe
Quoted 25 lines · Håkon Bugge
On RoCE systems, a CM REQ contains a Primary Hop Limit > 1 and Primary Subnet Local is zero.
In
cm_req_handler(), thecm_process_routed_req()function is called. Since the Primary Subnet Local value is zero in the request, and since this is RoCE (Primary Local LID is permissive), the following statement will be executed:IBA_SET(CM_REQ_PRIMARY_SL, req_msg, wc->sl);This corrupts SL in
req_msgif it was different from zero. In other words, a request to setup a connection using an SL != zero, will not be honored, and a connection using SL zero will be created instead.Fixed by not calling
cm_process_routed_req()on RoCE systems.Fixes 3971c9f6dbf2 ("IB/cm: Add interim support for routed paths")Signed-off-by Håkon Bugge <[email protected]>
drivers/infiniband/core/cm.c| 3 ++-1 file changed, 2 insertions(+), 1 deletion(-)drivers/infiniband/core/cm.c +1
-0+++ b/drivers/infiniband/core/cm.cdrivers/infiniband/core/cm.c static int cm_req_handler(struct cm_work *work) · line 2142
Quoted 6 lines
drivers/infiniband/core/cm.c +2
-1@@ -2138,7 +2138,8 @@ static int cm_req_handler(struct cm_work *work) goto destroy; } -cm_process_routed_req (req_msg, work->mad_recv_wc->wc); +if (cm_id_priv->av.ah_attr.type !=RDMA_AH_ATTR_TYPE_ROCE ) +cm_process_routed_req (req_msg, work->mad_recv_wc->wc);why use ah_attr.type when a few lines below we have:
Quoted 7 lines
if (gid_attr && rdma_protocol_roce(work->port->cm_dev->ib_device, work->port->port_num)) {?
I suspect you can just move this into the else?
I can counter that by saying ah_attr.type is used ~10 lines further down in the conditional call to sa_path_set_dmac() ;-)
Further, in
Quoted 3 lines
if (gid_attr &&
rdma_protocol_roce(work->port->cm_dev->ib_device,
work->port->port_num)) {I cannot really see how gid_attr could be null. If ib_init_ah_attr_from_wc() succeeds, it is set after the call to cm_init_av_for_response() above. May be using ah_attr.type in this test instead, for uniformity and readability?
I have no strong opinion.
Let me know your preference.
Thxs, Håkon
Quoted 58 lines · Håkon Bugge
Quoted 40 lines · Jason Gunthorpe
Quoted 25 lines · Håkon Bugge
On RoCE systems, a CM REQ contains a Primary Hop Limit > 1 and Primary Subnet Local is zero.
In
cm_req_handler(), thecm_process_routed_req()function is called. Since the Primary Subnet Local value is zero in the request, and since this is RoCE (Primary Local LID is permissive), the following statement will be executed:IBA_SET(CM_REQ_PRIMARY_SL, req_msg, wc->sl);This corrupts SL in
req_msgif it was different from zero. In other words, a request to setup a connection using an SL != zero, will not be honored, and a connection using SL zero will be created instead.Fixed by not calling
cm_process_routed_req()on RoCE systems.Fixes 3971c9f6dbf2 ("IB/cm: Add interim support for routed paths")Signed-off-by Håkon Bugge <[email protected]>
drivers/infiniband/core/cm.c| 3 ++-1 file changed, 2 insertions(+), 1 deletion(-)drivers/infiniband/core/cm.c +1
-0+++ b/drivers/infiniband/core/cm.cdrivers/infiniband/core/cm.c static int cm_req_handler(struct cm_work *work) · line 2142
Quoted 6 lines
drivers/infiniband/core/cm.c +2
-1@@ -2138,7 +2138,8 @@ static int cm_req_handler(struct cm_work *work) goto destroy; } -cm_process_routed_req (req_msg, work->mad_recv_wc->wc); +if (cm_id_priv->av.ah_attr.type !=RDMA_AH_ATTR_TYPE_ROCE ) +cm_process_routed_req (req_msg, work->mad_recv_wc->wc);why use ah_attr.type when a few lines below we have:
Quoted 7 lines
if (gid_attr && rdma_protocol_roce(work->port->cm_dev->ib_device, work->port->port_num)) {?
I suspect you can just move this into the else?
I can counter that by saying ah_attr.type is used ~10 lines further down in the conditional call to
sa_path_set_dmac();-)Further, in
Quoted 3 lines
if (gid_attr && rdma_protocol_roce(work->port->cm_dev->ib_device, work->port->port_num)) {
Quoted 5 lines
I cannot really see how
gid_attrcould be null. Ifib_init_ah_attr_from_wc()succeeds, it is set after the call tocm_init_av_for_response()above. May be using ah_attr.type in this test instead, for uniformity and readability?I have no strong opinion.
Let me know your preference.
A gentle ping.
Thxs, Håkon
Quoted 3 lines
Thxs, Håkon
Quoted 43 lines · Håkon Bugge
Quoted 43 lines · Jason Gunthorpe
Quoted 25 lines · Håkon Bugge
On RoCE systems, a CM REQ contains a Primary Hop Limit > 1 and Primary Subnet Local is zero.
In
cm_req_handler(), thecm_process_routed_req()function is called. Since the Primary Subnet Local value is zero in the request, and since this is RoCE (Primary Local LID is permissive), the following statement will be executed:IBA_SET(CM_REQ_PRIMARY_SL, req_msg, wc->sl);This corrupts SL in
req_msgif it was different from zero. In other words, a request to setup a connection using an SL != zero, will not be honored, and a connection using SL zero will be created instead.Fixed by not calling
cm_process_routed_req()on RoCE systems.Fixes 3971c9f6dbf2 ("IB/cm: Add interim support for routed paths")Signed-off-by Håkon Bugge <[email protected]>
drivers/infiniband/core/cm.c| 3 ++-1 file changed, 2 insertions(+), 1 deletion(-)drivers/infiniband/core/cm.c +1
-0+++ b/drivers/infiniband/core/cm.cdrivers/infiniband/core/cm.c static int cm_req_handler(struct cm_work *work) · line 2142
Quoted 6 lines
drivers/infiniband/core/cm.c +2
-1@@ -2138,7 +2138,8 @@ static int cm_req_handler(struct cm_work *work) goto destroy; } -cm_process_routed_req (req_msg, work->mad_recv_wc->wc); +if (cm_id_priv->av.ah_attr.type !=RDMA_AH_ATTR_TYPE_ROCE ) +cm_process_routed_req (req_msg, work->mad_recv_wc->wc);why use ah_attr.type when a few lines below we have:
if (gid_attr && rdma_protocol_roce(work->port->cm_dev->ib_device, work->port->port_num)) {
Quoted 6 lines
Quoted 3 lines
?
I suspect you can just move this into the else?
I can counter that by saying ah_attr.type is used ~10 lines further down in the conditional call to
sa_path_set_dmac();-)
Hum, OK. Please send an additional patch to unify everything around av.ah_attr.type
Quoted 8 lines
Quoted 3 lines
if (gid_attr && rdma_protocol_roce(work->port->cm_dev->ib_device, work->port->port_num)) {I cannot really see how
gid_attrcould be null. Ifib_init_ah_attr_from_wc()succeeds, it is set after the call tocm_init_av_for_response()above. May be using ah_attr.type in this test instead, for uniformity and readability?
The GRH is optional, ib_init_ah_attr_from_wc() only sets it conditionally.
Applied to for-next
Thanks,
Jason
Quoted 50 lines · Jason Gunthorpe
Quoted 45 lines · Håkon Bugge
Quoted 45 lines · Jason Gunthorpe
Quoted 25 lines · Håkon Bugge
On RoCE systems, a CM REQ contains a Primary Hop Limit > 1 and Primary Subnet Local is zero.
In
cm_req_handler(), thecm_process_routed_req()function is called. Since the Primary Subnet Local value is zero in the request, and since this is RoCE (Primary Local LID is permissive), the following statement will be executed:IBA_SET(CM_REQ_PRIMARY_SL, req_msg, wc->sl);This corrupts SL in
req_msgif it was different from zero. In other words, a request to setup a connection using an SL != zero, will not be honored, and a connection using SL zero will be created instead.Fixed by not calling
cm_process_routed_req()on RoCE systems.Fixes 3971c9f6dbf2 ("IB/cm: Add interim support for routed paths")Signed-off-by Håkon Bugge <[email protected]>
drivers/infiniband/core/cm.c| 3 ++-1 file changed, 2 insertions(+), 1 deletion(-)drivers/infiniband/core/cm.c +1
-0+++ b/drivers/infiniband/core/cm.cdrivers/infiniband/core/cm.c static int cm_req_handler(struct cm_work *work) · line 2142
Quoted 6 lines
drivers/infiniband/core/cm.c +2
-1@@ -2138,7 +2138,8 @@ static int cm_req_handler(struct cm_work *work) goto destroy; } -cm_process_routed_req (req_msg, work->mad_recv_wc->wc); +if (cm_id_priv->av.ah_attr.type !=RDMA_AH_ATTR_TYPE_ROCE ) +cm_process_routed_req (req_msg, work->mad_recv_wc->wc);why use ah_attr.type when a few lines below we have:
if (gid_attr && rdma_protocol_roce(work->port->cm_dev->ib_device, work->port->port_num)) {?
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
I suspect you can just move this into the else?
I can counter that by saying ah_attr.type is used ~10 lines further down in the conditional call to
sa_path_set_dmac();-)Hum, OK. Please send an additional patch to unify everything around av.ah_attr.type
Will do.
Quoted 11 lines
Quoted 8 lines
Quoted 3 lines
if (gid_attr && rdma_protocol_roce(work->port->cm_dev->ib_device, work->port->port_num)) {I cannot really see how
gid_attrcould be null. Ifib_init_ah_attr_from_wc()succeeds, it is set after the call tocm_init_av_for_response()above. May be using ah_attr.type in this test instead, for uniformity and readability?The GRH is optional,
ib_init_ah_attr_from_wc()only sets it conditionally.
True. But one of the conditions to set sgid_attr is rdma_protocol_roce(). Hence the first term in:
if (gid_attr && rdma_protocol_roce())
is superfluous. This because, it cannot be NULL on RoCE systems, because it is dereferenced in:
cm_init_av_for_response()
ib_init_ah_attr_from_wc()
rdma_move_grh_sgid_attr()I'll send the patch with the gid_attr term and let you can decide.
Thxs, Håkon
Quoted 5 lines
Applied to for-next
Thanks,
Jason