RDMA/siw: Fix setting active_{speed, width} attributes
Subsystems (2)
Make sure to set the active_{speed, width} attributes to avoid reporting the same values regardless of the underlying device.
--- drivers/infiniband/sw/siw/siw_verbs.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)
drivers/infiniband/sw/siw/siw_verbs.c +4 -3
@@ -165,11 +165,12 @@ int siw_query_port(struct ib_device *base_dev, u8 port, struct ib_port_attr *attr) {struct siw_device *sdev =to_siw_dev (base_dev); +int rc;memset (attr,0 ,sizeof (*attr)); - attr->active_speed =2 ; - attr->active_width =2 ; + rc =ib_get_eth_speed (base_dev, port, &attr->active_speed, + &attr->active_width); attr->gid_tbl_len =1 ; attr->max_msg_sz =-1 ; attr->max_mtu =ib_mtu_int_to_enum (sdev->netdev->mtu);
@@ -192,7 +193,7 @@ int siw_query_port(struct ib_device *base_dev, u8 port,* attr->subnet_timeout = 0; * attr->init_type_repy = 0; */ -return 0 ; +return rc; } int siw_get_port_immutable(struct ib_device *base_dev, u8 port,
Signature · 1 line
2.21.1
-----"Kamal Heib" <[email protected]> wrote: -----
Quoted 49 lines
To: [email protected]
From: "Kamal Heib" <[email protected]>
Date: 02/13/2020 02:07PM
Cc: "Jason Gunthorpe" <[email protected]>, "Doug Ledford" <[email protected]>, "Bernard Metzler" <[email protected]>, "Kamal Heib" <[email protected]>
Subject: [EXTERNAL] [PATH for-next] RDMA/siw: Fix setting active_{speed, width} attributesMake sure to set the active_{speed, width} attributes to avoid reporting
the same values regardless of the underlying device.Fixes 303ae1cdfdf7 ("rdma/siw: application interface")Signed-off-by Kamal Heib <[email protected]>---
drivers/infiniband/sw/siw/siw_verbs.c| 7 ++++---1 file changed, 4 insertions(+), 3 deletions(-)drivers/infiniband/sw/siw/siw_verbs.c +3
-2@@ -165,11 +165,12 @@ int siw_query_port(struct ib_device *base_dev,u8 port, struct ib_port_attr *attr) { struct siw_device *sdev = to_siw_dev(base_dev); +int rc;memset (attr,0 ,sizeof (*attr)); - attr->active_speed =2 ; - attr->active_width =2 ; + rc =ib_get_eth_speed (base_dev, port, &attr->active_speed, + &attr->active_width); attr->gid_tbl_len =1 ; attr->max_msg_sz =-1 ;attr->max_mtu = ib_mtu_int_to_enum(sdev->netdev->mtu);drivers/infiniband/sw/siw/siw_verbs.c +1
-1@@ -192,7 +193,7 @@ int siw_query_port(struct ib_device *base_dev, u8port, * attr->subnet_timeout = 0; * attr->init_type_repy = 0; */ -return 0 ; +return rc; }
Quoted 5 lines
int siw_get_port_immutable(struct
ib_device*base_dev, u8 port,Signature · 1 line
2.21.1
Hi Kamal,
Many thanks for looking after this! So there definitely seem to be applications which are taking care of those values. So, good to get my obvious laziness fixed.
I tried your patch on a 40Gbs Ethernet link (Chelsio cxgb4 driver). Works in principle, but reported numbers are off. I am not saying I would get right numbers when using Chelsio HW iWarp (iw_cxgb4), but it's closer to reality (using ibv_devinfo <ibname> -vv)
iw_cxgb4 driver:
...
active_width: 4X (2)
active_speed: 25.0 Gbps (32)
siw driver with your patch:
...
active_width: 4X (2)
active_speed: 10.0 Gbps (8)
Any idea how we can improve that, maybe coming even close to reality (40Gbs)?
Another remark: It has been siw folklore to name integer return values 'rv', and not 'rc'. I never liked 'return code'. It's a value in principle, sometimes it's interpreted as a code though, as in your case.
Many thanks!
Bernard.
Quoted 51 lines · Bernard Metzler
-----"Kamal Heib" <[email protected]> wrote: -----
Quoted 49 lines
To: [email protected]
From: "Kamal Heib" <[email protected]>
Date: 02/13/2020 02:07PM
Cc: "Jason Gunthorpe" <[email protected]>, "Doug Ledford" <[email protected]>, "Bernard Metzler" <[email protected]>, "Kamal Heib" <[email protected]>
Subject: [EXTERNAL] [PATH for-next] RDMA/siw: Fix setting active_{speed, width} attributesMake sure to set the active_{speed, width} attributes to avoid reporting
the same values regardless of the underlying device.Fixes 303ae1cdfdf7 ("rdma/siw: application interface")Signed-off-by Kamal Heib <[email protected]>---
drivers/infiniband/sw/siw/siw_verbs.c| 7 ++++---1 file changed, 4 insertions(+), 3 deletions(-)drivers/infiniband/sw/siw/siw_verbs.c +3
-2@@ -165,11 +165,12 @@ int siw_query_port(struct ib_device *base_dev,u8 port, struct ib_port_attr *attr) { struct siw_device *sdev = to_siw_dev(base_dev); +int rc;memset (attr,0 ,sizeof (*attr)); - attr->active_speed =2 ; - attr->active_width =2 ; + rc =ib_get_eth_speed (base_dev, port, &attr->active_speed, + &attr->active_width); attr->gid_tbl_len =1 ; attr->max_msg_sz =-1 ;attr->max_mtu = ib_mtu_int_to_enum(sdev->netdev->mtu);drivers/infiniband/sw/siw/siw_verbs.c +1
-1@@ -192,7 +193,7 @@ int siw_query_port(struct ib_device *base_dev, u8port, * attr->subnet_timeout = 0; * attr->init_type_repy = 0; */ -return 0 ; +return rc; }
Quoted 6 lines
Quoted 5 lines
int siw_get_port_immutable(struct
ib_device*base_dev, u8 port,Signature · 1 line
2.21.1Hi Kamal,
Hi Bernard,
Quoted 4 lines
Many thanks for looking after this! So there definitely seem to be applications which are taking care of those values. So, good to get my obvious laziness fixed.
Sure :)
Quoted 10 lines
I tried your patch on a 40Gbs Ethernet link (Chelsio cxgb4 driver). Works in principle, but reported numbers are off. I am not saying I would get right numbers when using Chelsio HW iWarp (
iw_cxgb4), but it's closer to reality (usingibv_devinfo<ibname> -vv)
iw_cxgb4driver:
...
active_width: 4X (2)
active_speed: 25.0 Gbps (32)
Quoted 7 lines
siw driver with your patch:
...
active_width: 4X (2)
active_speed: 10.0 Gbps (8)Any idea how we can improve that, maybe coming even close to reality (40Gbs)?
Could you please share the output of ethtool <if_name> for the underlying net device that used for both iw_cxgb4 and siw?
Quoted 6 lines
Another remark: It has been siw folklore to name integer return values 'rv', and not 'rc'. I never liked 'return code'. It's a value in principle, sometimes it's interpreted as a code though, as in your case.
Sure, I'll fix it in v2.
Quoted 4 lines
Many thanks!
Bernard.
Thanks,
Kamal
-----"Kamal Heib" <[email protected]> wrote: -----
Quoted 94 lines
To: "Bernard Metzler" <[email protected]>
From: "Kamal Heib" <[email protected]>
Date: 02/16/2020 02:43PM
Cc: [email protected], "Jason Gunthorpe" <[email protected]>, "Doug Ledford" <[email protected]>
Subject: [EXTERNAL] Re: [PATH for-next] RDMA/siw: Fix setting active_{speed, width} attributesQuoted 7 lines · Bernard Metzler
-----"Kamal Heib" <[email protected]> wrote: -----
Quoted 5 lines
To: [email protected]
From: "Kamal Heib" <[email protected]>
Date: 02/13/2020 02:07PM
Cc: "Jason Gunthorpe" <[email protected]>, "Doug Ledford" <[email protected]>, "Bernard Metzler" <[email protected]>,"Kamal
Quoted 14 lines
Quoted 14 lines
Heib" <[email protected]>
Subject: [EXTERNAL] [PATH for-next] RDMA/siw: Fix setting active_{speed, width} attributesMake sure to set the active_{speed, width} attributes to avoid reporting
the same values regardless of the underlying device.Fixes 303ae1cdfdf7 ("rdma/siw: application interface")Signed-off-by Kamal Heib <[email protected]>---
drivers/infiniband/sw/siw/siw_verbs.c| 7 ++++---1 file changed, 4 insertions(+), 3 deletions(-)Quoted 6 lines
drivers/infiniband/sw/siw/siw_verbs.c int siw_query_port(struct ib_device
Quoted 6 lines
drivers/infiniband/sw/siw/siw_verbs.c +0
-0@@ -165,11 +165,12 @@ int siw_query_port(struct ib_device
*base_dev,Quoted 8 lines
Quoted 8 lines
drivers/infiniband/sw/siw/siw_verbs.c +1
-0u8 port, struct ib_port_attr *attr) { struct siw_device *sdev = to_siw_dev(base_dev); +int rc;memset (attr,0 ,sizeof (*attr));Quoted 8 lines
drivers/infiniband/sw/siw/siw_verbs.c int siw_query_port(struct ib_device *base_dev,
Quoted 8 lines
drivers/infiniband/sw/siw/siw_verbs.c +2
-2- attr->active_speed =2 ; - attr->active_width =2 ; + rc =ib_get_eth_speed (base_dev, port, &attr->active_speed, + &attr->active_width); attr->gid_tbl_len =1 ; attr->max_msg_sz =-1 ;attr->max_mtu = ib_mtu_int_to_enum(sdev->netdev->mtu);drivers/infiniband/sw/siw/siw_verbs.c +0
-0@@ -192,7 +193,7 @@ int siw_query_port(struct ib_device *base_dev,u8
Quoted 8 lines
Quoted 8 lines
drivers/infiniband/sw/siw/siw_verbs.c +1
-1port, * attr->subnet_timeout = 0; * attr->init_type_repy = 0; */ -return 0 ; +return rc; }Quoted 6 lines
Quoted 5 lines
int siw_get_port_immutable(struct
ib_device*base_dev, u8 port,Signature · 1 line
2.21.1Hi Kamal,
Hi Bernard,
Quoted 4 lines
Many thanks for looking after this! So there definitely seem to be applications which are taking care of those values. So, good to get my obvious laziness fixed.
Sure :)
Quoted 15 lines
I tried your patch on a 40Gbs Ethernet link (Chelsio cxgb4 driver). Works in principle, but reported numbers are off. I am not saying I would get right numbers when using Chelsio HW iWarp (
iw_cxgb4), but it's closer to reality (usingibv_devinfo<ibname> -vv)
iw_cxgb4driver:
...
active_width: 4X (2)
active_speed: 25.0 Gbps (32)siw driver with your patch:
...
active_width: 4X (2)
active_speed: 10.0 Gbps (8)
Quoted 7 lines
Quoted 2 lines
Any idea how we can improve that, maybe coming even close to reality (40Gbs)?
Could you please share the output of ethtool <
if_name> for the underlying
net device that used for bothiw_cxgb4and siw?
H Kamal,
Sure! Speed looks correct, and its also what I get at maximum:
[bmt@spoke ~]$ ethtool enp1s0f4
Settings for enp1s0f4:
Supported ports: [ FIBRE ]
Supported link modes: 40000baseSR4/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: Yes
Supported FEC modes: None
Advertised link modes: 40000baseSR4/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: Yes
Advertised FEC modes: None
Link partner advertised link modes: 40000baseSR4/Full Link partner advertised pause frame use: Symmetric
Link partner advertised auto-negotiation: Yes
Link partner advertised FEC modes: None
Speed: 40000Mb/s
Duplex: Full
Port: Direct Attach Copper
PHYAD: 255
Transceiver: internal
Auto-negotiation: on
Cannot get wake-on-lan settings: Operation not permitted
Current message level: 0x000000ff (255)
drv probe link timer ifdown ifup rx_err tx_err
Link detected: yes
Quoted 125 lines · Bernard Metzler
-----"Kamal Heib" <[email protected]> wrote: -----
Quoted 94 lines
To: "Bernard Metzler" <[email protected]>
From: "Kamal Heib" <[email protected]>
Date: 02/16/2020 02:43PM
Cc: [email protected], "Jason Gunthorpe" <[email protected]>, "Doug Ledford" <[email protected]>
Subject: [EXTERNAL] Re: [PATH for-next] RDMA/siw: Fix setting active_{speed, width} attributesQuoted 7 lines · Bernard Metzler
-----"Kamal Heib" <[email protected]> wrote: -----
Quoted 5 lines
To: [email protected]
From: "Kamal Heib" <[email protected]>
Date: 02/13/2020 02:07PM
Cc: "Jason Gunthorpe" <[email protected]>, "Doug Ledford" <[email protected]>, "Bernard Metzler" <[email protected]>,"Kamal
Quoted 14 lines
Quoted 14 lines
Heib" <[email protected]>
Subject: [EXTERNAL] [PATH for-next] RDMA/siw: Fix setting active_{speed, width} attributesMake sure to set the active_{speed, width} attributes to avoid reporting
the same values regardless of the underlying device.Fixes 303ae1cdfdf7 ("rdma/siw: application interface")Signed-off-by Kamal Heib <[email protected]>---
drivers/infiniband/sw/siw/siw_verbs.c| 7 ++++---1 file changed, 4 insertions(+), 3 deletions(-)Quoted 6 lines
drivers/infiniband/sw/siw/siw_verbs.c int siw_query_port(struct ib_device
Quoted 6 lines
drivers/infiniband/sw/siw/siw_verbs.c +0
-0@@ -165,11 +165,12 @@ int siw_query_port(struct ib_device
*base_dev,Quoted 8 lines
Quoted 8 lines
drivers/infiniband/sw/siw/siw_verbs.c +1
-0u8 port, struct ib_port_attr *attr) { struct siw_device *sdev = to_siw_dev(base_dev); +int rc;memset (attr,0 ,sizeof (*attr));Quoted 8 lines
drivers/infiniband/sw/siw/siw_verbs.c int siw_query_port(struct ib_device *base_dev,
Quoted 8 lines
drivers/infiniband/sw/siw/siw_verbs.c +2
-2- attr->active_speed =2 ; - attr->active_width =2 ; + rc =ib_get_eth_speed (base_dev, port, &attr->active_speed, + &attr->active_width); attr->gid_tbl_len =1 ; attr->max_msg_sz =-1 ;attr->max_mtu = ib_mtu_int_to_enum(sdev->netdev->mtu);drivers/infiniband/sw/siw/siw_verbs.c +0
-0@@ -192,7 +193,7 @@ int siw_query_port(struct ib_device *base_dev,u8
Quoted 8 lines
Quoted 8 lines
drivers/infiniband/sw/siw/siw_verbs.c +1
-1port, * attr->subnet_timeout = 0; * attr->init_type_repy = 0; */ -return 0 ; +return rc; }Quoted 6 lines
Quoted 5 lines
int siw_get_port_immutable(struct
ib_device*base_dev, u8 port,Signature · 1 line
2.21.1Hi Kamal,
Hi Bernard,
Quoted 4 lines
Many thanks for looking after this! So there definitely seem to be applications which are taking care of those values. So, good to get my obvious laziness fixed.
Sure :)
Quoted 15 lines
I tried your patch on a 40Gbs Ethernet link (Chelsio cxgb4 driver). Works in principle, but reported numbers are off. I am not saying I would get right numbers when using Chelsio HW iWarp (
iw_cxgb4), but it's closer to reality (usingibv_devinfo<ibname> -vv)
iw_cxgb4driver:
...
active_width: 4X (2)
active_speed: 25.0 Gbps (32)siw driver with your patch:
...
active_width: 4X (2)
active_speed: 10.0 Gbps (8)Quoted 7 lines
Quoted 2 lines
Any idea how we can improve that, maybe coming even close to reality (40Gbs)?
Could you please share the output of ethtool <
if_name> for the underlying
net device that used for bothiw_cxgb4and siw?H Kamal,
Sure! Speed looks correct, and its also what I get at maximum:
[bmt@spoke ~]$ ethtool enp1s0f4
Settings for enp1s0f4:
Supported ports: [ FIBRE ]
Supported link modes: 40000baseSR4/Full
Supported pause frame use: Symmetric Supports auto-negotiation: Yes
Supported FEC modes: None
Advertised link modes: 40000baseSR4/Full Advertised pause frame use: Symmetric Advertised auto-negotiation: Yes
Advertised FEC modes: None
Link partner advertised link modes: 40000baseSR4/Full Link partner advertised pause frame use: Symmetric Link partner advertised auto-negotiation: Yes Link partner advertised FEC modes: None
Speed: 40000Mb/s
Duplex: Full
Quoted 9 lines
Port: Direct Attach Copper
PHYAD: 255
Transceiver: internal
Auto-negotiation: on
Cannot get wake-on-lan settings: Operation not permitted
Current message level:0x000000ff(255)
drv probe link timer ifdown ifuprx_errtx_err
Link detected: yes
Hi Bernard,
Well, this is the expected value for 40GbE, Because the reported value is 4X aggregation of FDR10. For more info please the table of speeds under [1].
[1] - https://en.wikipedia.org/wiki/InfiniBand
Thanks,
Kamal
-----"Kamal Heib" <[email protected]> wrote: -----
Quoted 157 lines
To: "Bernard Metzler" <[email protected]>
From: "Kamal Heib" <[email protected]>
Date: 02/17/2020 03:12PM
Cc: [email protected], "Jason Gunthorpe" <[email protected]>, "Doug Ledford" <[email protected]>
Subject: [EXTERNAL] Re: [PATH for-next] RDMA/siw: Fix setting active_{speed, width} attributesQuoted 48 lines · Bernard Metzler
-----"Kamal Heib" <[email protected]> wrote: -----
Quoted 46 lines
To: "Bernard Metzler" <[email protected]>
From: "Kamal Heib" <[email protected]>
Date: 02/16/2020 02:43PM
Cc: [email protected], "Jason Gunthorpe" <[email protected]>, "Doug Ledford" <[email protected]>
Subject: [EXTERNAL] Re: [PATH for-next] RDMA/siw: Fix setting active_{speed, width} attributesQuoted 7 lines · Bernard Metzler
-----"Kamal Heib" <[email protected]> wrote: -----
Quoted 5 lines
To: [email protected]
From: "Kamal Heib" <[email protected]>
Date: 02/13/2020 02:07PM
Cc: "Jason Gunthorpe" <[email protected]>, "Doug Ledford" <[email protected]>, "Bernard Metzler" <[email protected]>,"Kamal
Quoted 14 lines
Quoted 14 lines
Heib" <[email protected]>
Subject: [EXTERNAL] [PATH for-next] RDMA/siw: Fix setting active_{speed, width} attributesMake sure to set the active_{speed, width} attributes to avoid reporting
the same values regardless of the underlying device.Fixes 303ae1cdfdf7 ("rdma/siw: application interface")Signed-off-by Kamal Heib <[email protected]>---
drivers/infiniband/sw/siw/siw_verbs.c| 7 ++++---1 file changed, 4 insertions(+), 3 deletions(-)Quoted 6 lines
drivers/infiniband/sw/siw/siw_verbs.c int siw_query_port(struct ib_device
Quoted 6 lines
drivers/infiniband/sw/siw/siw_verbs.c +0
-0@@ -165,11 +165,12 @@ int siw_query_port(struct ib_device
*base_dev,Quoted 8 lines
Quoted 8 lines
drivers/infiniband/sw/siw/siw_verbs.c +1
-0u8 port, struct ib_port_attr *attr) { struct siw_device *sdev = to_siw_dev(base_dev); +int rc;memset (attr,0 ,sizeof (*attr));Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
drivers/infiniband/sw/siw/siw_verbs.c int siw_query_port(struct ib_device
Quoted 8 lines
drivers/infiniband/sw/siw/siw_verbs.c +2
-2- attr->active_speed =2 ; - attr->active_width =2 ; + rc =ib_get_eth_speed (base_dev, port, &attr->active_speed, + &attr->active_width); attr->gid_tbl_len =1 ; attr->max_msg_sz =-1 ;attr->max_mtu = ib_mtu_int_to_enum(sdev->netdev->mtu);drivers/infiniband/sw/siw/siw_verbs.c +0
-0@@ -192,7 +193,7 @@ int siw_query_port(struct ib_device
*base_dev,Quoted 10 lines
Quoted 10 lines
u8
Quoted 9 lines
Quoted 9 lines
drivers/infiniband/sw/siw/siw_verbs.c +1
-1port, * attr->subnet_timeout = 0; * attr->init_type_repy = 0; */ -return 0 ; +return rc; }int siw_get_port_immutable(struct
ib_device*base_dev, u8port,
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Quoted 4 lines
Signature · 1 line
2.21.1Hi Kamal,
Hi Bernard,
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
Many thanks for looking after this! So there definitely seem to be applications which are taking care of those values. So, good to get my obvious laziness fixed.
Sure :)
Quoted 1 line
I tried your patch on a 40Gbs Ethernet link (Chelsio cxgb4
driver).
Quoted 1 line
Quoted 1 line
Quoted 1 line
Works in principle, but reported numbers are off. I am not
saying
Quoted 1 line
Quoted 1 line
Quoted 1 line
I would get right numbers when using Chelsio HW iWarp
(
iw_cxgb4),Quoted 40 lines
Quoted 12 lines
Quoted 12 lines
but it's closer to reality (using
ibv_devinfo<ibname> -vv)
iw_cxgb4driver:
...
active_width: 4X (2)
active_speed: 25.0 Gbps (32)siw driver with your patch:
...
active_width: 4X (2)
active_speed: 10.0 Gbps (8)Quoted 7 lines
Quoted 2 lines
Any idea how we can improve that, maybe coming even close to reality (40Gbs)?
Could you please share the output of ethtool <
if_name> for the underlying
net device that used for bothiw_cxgb4and siw?H Kamal,
Sure! Speed looks correct, and its also what I get at maximum:
[bmt@spoke ~]$ ethtool enp1s0f4
Settings for enp1s0f4:
Supported ports: [ FIBRE ]
Supported link modes: 40000baseSR4/Full
Supported pause frame use: Symmetric Supports auto-negotiation: Yes
Supported FEC modes: None
Advertised link modes: 40000baseSR4/Full Advertised pause frame use: Symmetric Advertised auto-negotiation: Yes
Advertised FEC modes: None
Link partner advertised link modes: 40000baseSR4/Full Link partner advertised pause frame use: Symmetric Link partner advertised auto-negotiation: Yes Link partner advertised FEC modes: None
Speed: 40000Mb/sQuoted 8 lines
Duplex: Full
Port: Direct Attach Copper
PHYAD: 255
Transceiver: internal
Auto-negotiation: on
Cannot get wake-on-lan settings: Operation not permitted
Current message level:0x000000ff(255)
drv probe link timer ifdown ifup
rx_errtx_errQuoted 2 lines
Link detected: yes
Hi Bernard,
Well, this is the expected value for 40GbE, Because the reported value
is 4X aggregation of FDR10. For more info please the table of speeds under [1].
Quoted 9 lines
[1] -
https://urldefense.proofpoint.com/v2/url?u=https-3A__en.wikipedia.org
_wiki_InfiniBand&d=DwIBAg&c=jf_iaSHvJObTbx-siA1ZOg&r=2TaYXQ0T-r8ZO1PP 1alNwU_QJcRRLfmYTAgd3QCvqSc&m=Ay2tYZ7Z-SHKRw8UZDCk76kwlZzvkXhRMrO_0jk YjcY&s=D4Z0CAH5UVO95WHNnUozLzrqjxRQgVe-2lc8_jwVlhw&e=Thanks,
Kamal
Hi Kamal, so I have to do the math! 4 x 10Gbs = 40Gbs. So it is all correct as reported by ibv_devinfio (and somehow strange what the iw_cxgb4 makes out of it).
Many thanks for the info,
Bernard.
Quoted 165 lines · Bernard Metzler
-----"Kamal Heib" <[email protected]> wrote: -----
Quoted 163 lines
To: "Bernard Metzler" <[email protected]>
From: "Kamal Heib" <[email protected]>
Date: 02/17/2020 03:12PM
Cc: [email protected], "Jason Gunthorpe" <[email protected]>, "Doug Ledford" <[email protected]>
Subject: [EXTERNAL] Re: [PATH for-next] RDMA/siw: Fix setting active_{speed, width} attributesQuoted 48 lines · Bernard Metzler
-----"Kamal Heib" <[email protected]> wrote: -----
Quoted 46 lines
To: "Bernard Metzler" <[email protected]>
From: "Kamal Heib" <[email protected]>
Date: 02/16/2020 02:43PM
Cc: [email protected], "Jason Gunthorpe" <[email protected]>, "Doug Ledford" <[email protected]>
Subject: [EXTERNAL] Re: [PATH for-next] RDMA/siw: Fix setting active_{speed, width} attributesQuoted 7 lines · Bernard Metzler
-----"Kamal Heib" <[email protected]> wrote: -----
Quoted 5 lines
To: [email protected]
From: "Kamal Heib" <[email protected]>
Date: 02/13/2020 02:07PM
Cc: "Jason Gunthorpe" <[email protected]>, "Doug Ledford" <[email protected]>, "Bernard Metzler" <[email protected]>,"Kamal
Quoted 14 lines
Quoted 14 lines
Heib" <[email protected]>
Subject: [EXTERNAL] [PATH for-next] RDMA/siw: Fix setting active_{speed, width} attributesMake sure to set the active_{speed, width} attributes to avoid reporting
the same values regardless of the underlying device.Fixes 303ae1cdfdf7 ("rdma/siw: application interface")Signed-off-by Kamal Heib <[email protected]>---
drivers/infiniband/sw/siw/siw_verbs.c| 7 ++++---1 file changed, 4 insertions(+), 3 deletions(-)Quoted 6 lines
drivers/infiniband/sw/siw/siw_verbs.c int siw_query_port(struct ib_device
Quoted 6 lines
drivers/infiniband/sw/siw/siw_verbs.c +0
-0@@ -165,11 +165,12 @@ int siw_query_port(struct ib_device
*base_dev,Quoted 8 lines
Quoted 8 lines
drivers/infiniband/sw/siw/siw_verbs.c +1
-0u8 port, struct ib_port_attr *attr) { struct siw_device *sdev = to_siw_dev(base_dev); +int rc;memset (attr,0 ,sizeof (*attr));Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
drivers/infiniband/sw/siw/siw_verbs.c int siw_query_port(struct ib_device
Quoted 8 lines
drivers/infiniband/sw/siw/siw_verbs.c +2
-2- attr->active_speed =2 ; - attr->active_width =2 ; + rc =ib_get_eth_speed (base_dev, port, &attr->active_speed, + &attr->active_width); attr->gid_tbl_len =1 ; attr->max_msg_sz =-1 ;attr->max_mtu = ib_mtu_int_to_enum(sdev->netdev->mtu);drivers/infiniband/sw/siw/siw_verbs.c +0
-0@@ -192,7 +193,7 @@ int siw_query_port(struct ib_device
*base_dev,Quoted 10 lines
Quoted 10 lines
u8
Quoted 9 lines
Quoted 9 lines
drivers/infiniband/sw/siw/siw_verbs.c +1
-1port, * attr->subnet_timeout = 0; * attr->init_type_repy = 0; */ -return 0 ; +return rc; }int siw_get_port_immutable(struct
ib_device*base_dev, u8port,
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Quoted 4 lines
Signature · 1 line
2.21.1Hi Kamal,
Hi Bernard,
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
Many thanks for looking after this! So there definitely seem to be applications which are taking care of those values. So, good to get my obvious laziness fixed.
Sure :)
Quoted 1 line
I tried your patch on a 40Gbs Ethernet link (Chelsio cxgb4
driver).
Quoted 1 line
Quoted 1 line
Quoted 1 line
Works in principle, but reported numbers are off. I am not
saying
Quoted 1 line
Quoted 1 line
Quoted 1 line
I would get right numbers when using Chelsio HW iWarp
(
iw_cxgb4),Quoted 40 lines
Quoted 12 lines
Quoted 12 lines
but it's closer to reality (using
ibv_devinfo<ibname> -vv)
iw_cxgb4driver:
...
active_width: 4X (2)
active_speed: 25.0 Gbps (32)siw driver with your patch:
...
active_width: 4X (2)
active_speed: 10.0 Gbps (8)Quoted 7 lines
Quoted 2 lines
Any idea how we can improve that, maybe coming even close to reality (40Gbs)?
Could you please share the output of ethtool <
if_name> for the underlying
net device that used for bothiw_cxgb4and siw?H Kamal,
Sure! Speed looks correct, and its also what I get at maximum:
[bmt@spoke ~]$ ethtool enp1s0f4
Settings for enp1s0f4:
Supported ports: [ FIBRE ]
Supported link modes: 40000baseSR4/Full
Supported pause frame use: Symmetric Supports auto-negotiation: Yes
Supported FEC modes: None
Advertised link modes: 40000baseSR4/Full Advertised pause frame use: Symmetric Advertised auto-negotiation: Yes
Advertised FEC modes: None
Link partner advertised link modes: 40000baseSR4/Full Link partner advertised pause frame use: Symmetric Link partner advertised auto-negotiation: Yes Link partner advertised FEC modes: None
Speed: 40000Mb/sQuoted 8 lines
Duplex: Full
Port: Direct Attach Copper
PHYAD: 255
Transceiver: internal
Auto-negotiation: on
Cannot get wake-on-lan settings: Operation not permitted
Current message level:0x000000ff(255)
drv probe link timer ifdown ifup
rx_errtx_errQuoted 2 lines
Link detected: yes
Hi Bernard,
Well, this is the expected value for 40GbE, Because the reported value
is 4X aggregation of FDR10. For more info please the table of speeds under [1].[1] -
https://urldefense.proofpoint.com/v2/url?u=https-3A__en.wikipedia.org
_wiki_InfiniBand&d=DwIBAg&c=jf_iaSHvJObTbx-siA1ZOg&r=2TaYXQ0T-r8ZO1PP 1alNwU_QJcRRLfmYTAgd3QCvqSc&m=Ay2tYZ7Z-SHKRw8UZDCk76kwlZzvkXhRMrO_0jk YjcY&s=D4Z0CAH5UVO95WHNnUozLzrqjxRQgVe-2lc8_jwVlhw&e=
Quoted 6 lines
Quoted 3 lines
Thanks,
KamalHi Kamal, so I have to do the math! 4 x 10Gbs = 40Gbs. So it is all correct as reported by
ibv_devinfio(and somehow strange what theiw_cxgb4makes out of it).
Hi Bernard,
Are you running siw and iw_cxgb4 on the same port or different ports? if not same, Is the port running iw_cxgb4 connected with 100G cable? If 40G cable is conencted to iw_cxgb4 as well then its reporting wrong.
Thanks.
Quoted 4 lines
Many thanks for the info,
Bernard.
-----"Potnuri Bharat Teja" <[email protected]> wrote: -----
Quoted 192 lines
To: "Bernard Metzler" <[email protected]>
From: "Potnuri Bharat Teja" <[email protected]>
Date: 02/17/2020 03:51PM
Cc: "Kamal Heib" <[email protected]>, "[email protected]" <[email protected]>, "Jason Gunthorpe" <[email protected]>, "Doug Ledford" <[email protected]>
Subject: [EXTERNAL] Re: [PATH for-next] RDMA/siw: Fix setting active_{speed, width} attributesQuoted 17 lines · Bernard Metzler
-----"Kamal Heib" <[email protected]> wrote: -----
Quoted 15 lines
To: "Bernard Metzler" <[email protected]>
From: "Kamal Heib" <[email protected]>
Date: 02/17/2020 03:12PM
Cc: [email protected], "Jason Gunthorpe" <[email protected]>, "Doug Ledford" <[email protected]>
Subject: [EXTERNAL] Re: [PATH for-next] RDMA/siw: Fix setting active_{speed, width} attributesQuoted 6 lines · Bernard Metzler
-----"Kamal Heib" <[email protected]> wrote: -----
Quoted 4 lines
To: "Bernard Metzler" <[email protected]>
From: "Kamal Heib" <[email protected]>
Date: 02/16/2020 02:43PM
Cc: [email protected], "Jason Gunthorpe"Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
"Doug Ledford" <[email protected]>
Subject: [EXTERNAL] Re: [PATH for-next] RDMA/siw: Fix setting active_{speed, width} attributesOn Thu, Feb 13, 2020 at 01:59:30PM +0000, Bernard Metzler
Quoted 7 lines · wrote:
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
-----"Kamal Heib" <[email protected]> wrote: -----
Quoted 5 lines
To: [email protected]
From: "Kamal Heib" <[email protected]>
Date: 02/13/2020 02:07PM
Cc: "Jason Gunthorpe" <[email protected]>, "Doug Ledford" <[email protected]>, "Bernard Metzler"Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
"Kamal
Quoted 5 lines
Quoted 5 lines
Heib" <[email protected]>
Subject: [EXTERNAL] [PATH for-next] RDMA/siw: Fix setting active_{speed, width} attributesMake sure to set the active_{speed, width} attributes to
avoid
Quoted 48 lines
Quoted 48 lines
Quoted 24 lines
Quoted 24 lines
Quoted 9 lines
Quoted 9 lines
reporting
the same values regardless of the underlying device.Fixes 303ae1cdfdf7 ("rdma/siw: application interface")Signed-off-by Kamal Heib <[email protected]>---
drivers/infiniband/sw/siw/siw_verbs.c| 7 ++++---1 file changed, 4 insertions(+), 3 deletions(-)Quoted 6 lines
drivers/infiniband/sw/siw/siw_verbs.c int siw_query_port(struct ib_device
Quoted 6 lines
drivers/infiniband/sw/siw/siw_verbs.c +0
-0@@ -165,11 +165,12 @@ int siw_query_port(struct ib_device
*base_dev,Quoted 8 lines
Quoted 8 lines
drivers/infiniband/sw/siw/siw_verbs.c +1
-0u8 port, struct ib_port_attr *attr) { struct siw_device *sdev = to_siw_dev(base_dev); +int rc;memset (attr,0 ,sizeof (*attr));Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
drivers/infiniband/sw/siw/siw_verbs.c int siw_query_port(struct ib_device
Quoted 8 lines
drivers/infiniband/sw/siw/siw_verbs.c +2
-2- attr->active_speed =2 ; - attr->active_width =2 ; + rc =ib_get_eth_speed (base_dev, port, &attr->active_speed, + &attr->active_width); attr->gid_tbl_len =1 ; attr->max_msg_sz =-1 ;attr->max_mtu = ib_mtu_int_to_enum(sdev->netdev->mtu);drivers/infiniband/sw/siw/siw_verbs.c +0
-0@@ -192,7 +193,7 @@ int siw_query_port(struct ib_device
*base_dev,Quoted 10 lines
Quoted 10 lines
u8
Quoted 9 lines
Quoted 9 lines
drivers/infiniband/sw/siw/siw_verbs.c +1
-1port, * attr->subnet_timeout = 0; * attr->init_type_repy = 0; */ -return 0 ; +return rc; }int siw_get_port_immutable(struct
ib_device*base_dev, u8port,
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Signature · 1 line
2.21.1Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
Hi Kamal,
Hi Bernard,
Quoted 1 line
Many thanks for looking after this! So there definitely seem
to
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
be applications which are taking care of those values. So,
good
Quoted 60 lines
Quoted 60 lines
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
to get my obvious laziness fixed.
Sure :)
Quoted 1 line
I tried your patch on a 40Gbs Ethernet link (Chelsio cxgb4
driver).
Quoted 1 line
Quoted 1 line
Quoted 1 line
Works in principle, but reported numbers are off. I am not
saying
Quoted 1 line
Quoted 1 line
Quoted 1 line
I would get right numbers when using Chelsio HW iWarp
(
iw_cxgb4),Quoted 40 lines
Quoted 12 lines
Quoted 12 lines
but it's closer to reality (using
ibv_devinfo<ibname> -vv)
iw_cxgb4driver:
...
active_width: 4X (2)
active_speed: 25.0 Gbps (32)siw driver with your patch:
...
active_width: 4X (2)
active_speed: 10.0 Gbps (8)Quoted 7 lines
Quoted 2 lines
Any idea how we can improve that, maybe coming even close to reality (40Gbs)?
Could you please share the output of ethtool <
if_name> for the underlying
net device that used for bothiw_cxgb4and siw?H Kamal,
Sure! Speed looks correct, and its also what I get at maximum:
[bmt@spoke ~]$ ethtool enp1s0f4
Settings for enp1s0f4:
Supported ports: [ FIBRE ]
Supported link modes: 40000baseSR4/Full
Supported pause frame use: Symmetric Supports auto-negotiation: Yes
Supported FEC modes: None
Advertised link modes: 40000baseSR4/Full Advertised pause frame use: Symmetric Advertised auto-negotiation: Yes
Advertised FEC modes: None
Link partner advertised link modes: 40000baseSR4/Full Link partner advertised pause frame use: Symmetric Link partner advertised auto-negotiation: Yes Link partner advertised FEC modes: None
Speed: 40000Mb/sQuoted 8 lines
Duplex: Full
Port: Direct Attach Copper
PHYAD: 255
Transceiver: internal
Auto-negotiation: on
Cannot get wake-on-lan settings: Operation not permitted
Current message level:0x000000ff(255)
drv probe link timer ifdown ifup
rx_errtx_errQuoted 8 lines
Quoted 8 lines
Quoted 2 lines
Link detected: yes
Hi Bernard,
Well, this is the expected value for 40GbE, Because the reported value
is 4X aggregation of FDR10. For more info please the table ofspeeds
Quoted 5 lines
Quoted 3 lines
under [1].
[1] -
https://urldefense.proofpoint.com/v2/url?u=https-3A__en.wikipedia.or
g
Quoted 2 lines
_wiki_InfiniBand&d=DwIBAg&c=jf_iaSHvJObTbx-siA1ZOg&r=2TaYXQ0T-r8ZO1P
P
Quoted 2 lines
1alNwU_QJcRRLfmYTAgd3QCvqSc&m=Ay2tYZ7Z-SHKRw8UZDCk76kwlZzvkXhRMrO_0j
k
Quoted 5 lines
Quoted 5 lines
YjcY&s=D4Z0CAH5UVO95WHNnUozLzrqjxRQgVe-2lc8_jwVlhw&e=
Thanks,
Kamal
Quoted 9 lines
Quoted 3 lines
Hi Kamal, so I have to do the math! 4 x 10Gbs = 40Gbs. So it is all correct as reported by
ibv_devinfio(and somehow strange what theiw_cxgb4makes out of it).Hi Bernard,
Are you running siw andiw_cxgb4on the same port or different ports? if not same, Is the port runningiw_cxgb4connected with 100G cable? If 40G cable is conencted toiw_cxgb4as well then its reporting wrong.
This is a single port 40Gbs setup. All my fault:
For iw_cxgb4, I looked at the reported values for the second port, which is not physically connected but reported PORT_DOWN and 4 x 25Gbs.
The connected port 1 has 4 x 10Gbs for both siw and iw_cxgb4. Sorry about the confusion!
So its all good.
Thanks
Bernard.