RDMA/ipoib: distribute cq completion vector better
PATCH 1/1 RDMA/ipoib: distribute cq completion vector better
Currently ipoib choose cq completion vector based on port number, when HCA only have one port, all the interface recv queue completion are bind to cq completion vector 0.
To better distribute the load, use same method as __ib_alloc_cq_any to choose completion vector, with the change, each interface now use different completion vectors.
--- drivers/infiniband/ulp/ipoib/ipoib_verbs.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/infiniband/ulp/ipoib/ipoib_verbs.c +2 -2
@@ -158,6 +158,7 @@ int ipoib_transport_dev_init(struct net_device *dev, struct ib_device *ca)int ret, size, req_vec;int i; +static atomic_t counter; size = ipoib_recvq_size +1 ; ret =ipoib_cm_dev_init (dev);
@@ -171,8 +172,7 @@ int ipoib_transport_dev_init(struct net_device *dev, struct ib_device *ca)if (ret != -EOPNOTSUPP )return ret; - req_vec = (priv->port -1 ) *2 ; - + req_vec =atomic_inc_return (&counter) *2 ; cq_attr.cqe = size; cq_attr.comp_vector = req_vec % priv->ca->num_comp_vectors; priv->recv_cq = ib_create_cq(priv->ca, ipoib_ib_rx_completion, NULL,
Signature · 1 line
2.25.1
Jason Gunthorpe
maintainerQuoted 8 lines · Jack Wang
Currently ipoib choose cq completion vector based on port number, when HCA only have one port, all the interface recv queue completion are bind to cq completion vector 0.
To better distribute the load, use same method as
__ib_alloc_cq_anyto choose completion vector, with the change, each interface now use different completion vectors.
Quoted 5 lines
Signed-off-by Jack Wang <[email protected]>Reviewed-by Gioh Kim <[email protected]>--- drivers/infiniband/ulp/ipoib/ipoib_verbs.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
If you care about IPoIB performance you should be using the accelerated IPoIB stuff, the drivers implementing that all provide much better handling of CQ affinity.
What scenario does this patch make a difference?
Jason
Quoted 16 lines · Jason Gunthorpe
Quoted 8 lines · Jack Wang
Currently ipoib choose cq completion vector based on port number, when HCA only have one port, all the interface recv queue completion are bind to cq completion vector 0.
To better distribute the load, use same method as
__ib_alloc_cq_anyto choose completion vector, with the change, each interface now use different completion vectors.Quoted 5 lines
Signed-off-by Jack Wang <[email protected]>Reviewed-by Gioh Kim <[email protected]>--- drivers/infiniband/ulp/ipoib/ipoib_verbs.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
Quoted 5 lines
If you care about IPoIB performance you should be using the accelerated IPoIB stuff, the drivers implementing that all provide much better handling of CQ affinity.
What scenario does this patch make a difference?
AFAIK the enhance mode is only for datagram mode, we are using connected mode.
Thanks!
Jason Gunthorpe
maintainerQuoted 16 lines · Jinpu Wang
Quoted 15 lines · Jason Gunthorpe
Quoted 12 lines · Jack Wang
Currently ipoib choose cq completion vector based on port number, when HCA only have one port, all the interface recv queue completion are bind to cq completion vector 0.
To better distribute the load, use same method as
__ib_alloc_cq_anyto choose completion vector, with the change, each interface now use different completion vectors.Signed-off-by Jack Wang <[email protected]>Reviewed-by Gioh Kim <[email protected]>drivers/infiniband/ulp/ipoib/ipoib_verbs.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
Quoted 7 lines
Quoted 5 lines
If you care about IPoIB performance you should be using the accelerated IPoIB stuff, the drivers implementing that all provide much better handling of CQ affinity.
What scenario does this patch make a difference?
AFAIK the enhance mode is only for datagram mode, we are using connected mode.
And you are using child interfaces or multiple cards?
Jason
Quoted 22 lines · Jason Gunthorpe
Quoted 20 lines · Jinpu Wang
Quoted 19 lines · Jason Gunthorpe
Quoted 4 lines · Jack Wang
Currently ipoib choose cq completion vector based on port number, when HCA only have one port, all the interface recv queue completion are bind to cq completion vector 0.
Quoted 8 lines
To better distribute the load, use same method as
__ib_alloc_cq_anyto choose completion vector, with the change, each interface now use different completion vectors.Signed-off-by Jack Wang <[email protected]>Reviewed-by Gioh Kim <[email protected]>drivers/infiniband/ulp/ipoib/ipoib_verbs.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)If you care about IPoIB performance you should be using the accelerated IPoIB stuff, the drivers implementing that all provide much better handling of CQ affinity.
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
What scenario does this patch make a difference?
AFAIK the enhance mode is only for datagram mode, we are using connected mode.
And you are using child interfaces or multiple cards?
we are using multiple child interfaces on Connect x5 HCA (MCX556A-ECAT) ibstat
CA 'mlx5_0'
CA type: MT4119
Number of ports: 1
Firmware version: 16.28.2006
Hardware version: 0
Node GUID: 0x98039b03006c7912
System image GUID: 0x98039b03006c7912
Port 1:
State: Active
Physical state: LinkUp
Rate: 40
Base lid: 38
LMC: 0
SM lid: 4
Capability mask: 0x2651e848
Port GUID: 0x98039b03006c7912
Link layer: InfiniBand
CA 'mlx5_1'
CA type: MT4119
Number of ports: 1
Firmware version: 16.28.2006
Hardware version: 0
Node GUID: 0x98039b03006c7913
System image GUID: 0x98039b03006c7912
Port 1:
State: Active
Physical state: LinkUp
Rate: 40
Base lid: 134
LMC: 0
SM lid: 224
Capability mask: 0x2651e848
Port GUID: 0x98039b03006c7913
Link layer: InfiniBand
Quoted 2 lines
Jason
Jason Gunthorpe
maintainerQuoted 23 lines · Jinpu Wang
Quoted 22 lines · Jason Gunthorpe
Quoted 20 lines · Jinpu Wang
Quoted 19 lines · Jason Gunthorpe
Quoted 4 lines · Jack Wang
Currently ipoib choose cq completion vector based on port number, when HCA only have one port, all the interface recv queue completion are bind to cq completion vector 0.
Quoted 8 lines
To better distribute the load, use same method as
__ib_alloc_cq_anyto choose completion vector, with the change, each interface now use different completion vectors.Signed-off-by Jack Wang <[email protected]>Reviewed-by Gioh Kim <[email protected]>drivers/infiniband/ulp/ipoib/ipoib_verbs.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)If you care about IPoIB performance you should be using the accelerated IPoIB stuff, the drivers implementing that all provide much better handling of CQ affinity.
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
What scenario does this patch make a difference?
AFAIK the enhance mode is only for datagram mode, we are using connected mode.
And you are using child interfaces or multiple cards?
we are using multiple child interfaces on Connect x5 HCA (MCX556A-ECAT)
Ok.. Do you think it would be better to change IPoIB to use the new shared CQ pool API?
Jason
Quoted 29 lines · Jason Gunthorpe
Quoted 27 lines · Jinpu Wang
Quoted 26 lines · Jason Gunthorpe
Quoted 23 lines · Jinpu Wang
Quoted 15 lines · Jason Gunthorpe
Quoted 4 lines · Jack Wang
Currently ipoib choose cq completion vector based on port number, when HCA only have one port, all the interface recv queue completion are bind to cq completion vector 0.
Quoted 8 lines
To better distribute the load, use same method as
__ib_alloc_cq_anyto choose completion vector, with the change, each interface now use different completion vectors.Signed-off-by Jack Wang <[email protected]>Reviewed-by Gioh Kim <[email protected]>drivers/infiniband/ulp/ipoib/ipoib_verbs.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)Quoted 5 lines
If you care about IPoIB performance you should be using the accelerated IPoIB stuff, the drivers implementing that all provide much better handling of CQ affinity.
What scenario does this patch make a difference?
AFAIK the enhance mode is only for datagram mode, we are using connected mode.
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
And you are using child interfaces or multiple cards?
we are using multiple child interfaces on Connect x5 HCA (MCX556A-ECAT)
Ok.. Do you think it would be better to change IPoIB to use the new shared CQ pool API?
Jason
I thought about it, but CQ pool API doesn't work with IB_POLL_DIRECT, ipoib is using NAPI,
and it's non-trivial to convert to CQ pool API. I'm not confident to do it properly,
Quoted 8 lines · Jack Wang
Currently ipoib choose cq completion vector based on port number, when HCA only have one port, all the interface recv queue completion are bind to cq completion vector 0.
To better distribute the load, use same method as
__ib_alloc_cq_anyto choose completion vector, with the change, each interface now use different completion vectors.
Quoted 5 lines
Signed-off-by Jack Wang <[email protected]>Reviewed-by Gioh Kim <[email protected]>--- drivers/infiniband/ulp/ipoib/ipoib_verbs.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
Applied to for next, thanks
Jason