mlx4: kernel 3.4-rc1 breaks libumad

Hi,

Apparently applications based on libumad can find local ports with kernel 3.2.x but not with kernel 3.4-rc1.

# uname -r
3.4.0-rc1
# ls /sys/class/infiniband/mlx4_0/ports/1/rate
/sys/class/infiniband/mlx4_0/ports/1/rate
# cat /sys/class/infiniband/mlx4_0/ports/1/rate
cat: /sys/class/infiniband/mlx4_0/ports/1/rate: Invalid argument

This breaks libumad because this makes sys_read_string() fail and hence also get_port().

Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

seems that this happens when the link layer is "wrong" e.g does /sys/class/infiniband/mlx4_0/ports/1/link_layer shows the actual link which is your hca port 1 is connected through? looking on this.

Or.

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

As far as I can see the link layer value is fine:

$ cat /sys/class/infiniband/mlx4_0/ports/1/link_layer
InfiniBand
$ cat /sys/class/infiniband/mlx4_0/ports/2/link_layer
InfiniBand

Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

So the two ports are actually connected to IB switch? please send ibv_devinfo output (HCA, firmware, etc).

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

The two ports are connected back-to-back to another mlx4 HCA. I noticed this behavior change since opensm stopped working after rebooting into 3.4-rc1. ibv_devinfo output is as follows:

# ibv_devinfo -v
hca_id: mlx4_0
        transport:                      InfiniBand (0)
        fw_ver:                         2.9.1000
        node_guid:                      0002:c903:0005:f34e
        sys_image_guid:                 0002:c903:0005:f351
        vendor_id:                      0x02c9
        vendor_part_id:                 26428
        hw_ver:                         0xA0
        board_id:                       MT_0BB0120003
        phys_port_cnt:                  2
        max_mr_size:                    0xffffffffffffffff
        page_size_cap:                  0xfffffe00
        max_qp:                         131008
        max_qp_wr:                      16384
        device_cap_flags:               0x007c9c76
        max_sge:                        32
        max_sge_rd:                     0
        max_cq:                         65408
        max_cqe:                        4194303
        max_mr:                         524272
        max_pd:                         32764
        max_qp_rd_atom:                 16
        max_ee_rd_atom:                 0
        max_res_rd_atom:                2096128
        max_qp_init_rd_atom:            128
        max_ee_init_rd_atom:            0
        atomic_cap:                     ATOMIC_HCA (1)
        max_ee:                         0
        max_rdd:                        0
        max_mw:                         0
        max_raw_ipv6_qp:                0
        max_raw_ethy_qp:                0
        max_mcast_grp:                  8192
        max_mcast_qp_attach:            248
        max_total_mcast_qp_attach:      2031616
        max_ah:                         0
        max_fmr:                        0
        max_srq:                        65472
        max_srq_wr:                     16383
        max_srq_sge:                    31
        max_pkeys:                      128
        local_ca_ack_delay:             15
                port:   1
                        state:                  PORT_ACTIVE (4)
                        max_mtu:                4096 (5)
                        active_mtu:             4096 (5)
                        sm_lid:                 1
                        port_lid:               1
                        port_lmc:               0x00
                        link_layer:             InfiniBand
                        max_msg_sz:             0x40000000
                        port_cap_flags:         0x0251086a
                        max_vl_num:             4 (3)
                        bad_pkey_cntr:          0x0
                        qkey_viol_cntr:         0x0
                        sm_sl:                  0
                        pkey_tbl_len:           128
                        gid_tbl_len:            128
                        subnet_timeout:         18
                        init_type_reply:        0
                        active_width:           4X (2)
                        active_speed:           10.0 Gbps (4)
                        phys_state:             LINK_UP (5)
                        GID[  0]:               fe80:0000:0000:0000:0002:c903:0005:f34f

                port:   2
                        state:                  PORT_INIT (2)
                        max_mtu:                4096 (5)
                        active_mtu:             4096 (5)
                        sm_lid:                 0
                        port_lid:               0
                        port_lmc:               0x00
                        link_layer:             InfiniBand
                        max_msg_sz:             0x40000000
                        port_cap_flags:         0x02510868
                        max_vl_num:             4 (3)
                        bad_pkey_cntr:          0x0
                        qkey_viol_cntr:         0x0
                        sm_sl:                  0
                        pkey_tbl_len:           128
                        gid_tbl_len:            128
                        subnet_timeout:         0
                        init_type_reply:        0
                        active_width:           4X (2)
                        active_speed:           10.0 Gbps (4)
                        phys_state:             LINK_UP (5)
                        GID[  0]:               fe80:0000:0000:0000:0002:c903:0005:f350

Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

can you add these prints and send me the output after attempting to cat the rate file?

Or.

okay, on a system which has IB on port 1 and Ethernet on port 2, using this patch I get these prints:

but if forcing port 2 link layer to be IB as well, which means we will land in ib_link_query_port for an Ethernet port, I get the below

So when doing the MAD_IFC port info query command on Ethernet port, the firmware returns the value of seven which isn't among the IB speeds and we are remained with rate=-1 in rate_show of drivers/infiniband/core/sysfs.c

It should be pretty simple to come with patch to that situation, but I want to better understand what happens on your system, waiting for the output...

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

libibumad (and infiniband-diags) are not yet RoCE ready AFAIK. Fixing that at least for libibumad is minor. Ira can comment on infiniband-diags.

I think there are 3 main issues here:
1. EINVAL can be returned from rate_show and hence "Invalid argument" rate string should be handled in libibumad. I think this was Bart's original point.
2. Why is rate_show returning EINVAL ? I think that's what you're trying to isolate with the additional printks you sent Bart for sysfs.c.
3. link_layer ethernet should also be handled which is the issue you raised.

-- Hal

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Bart,

Would you please try libibumad patch below ? Thanks.

-- Hal

libbibumad/umad.c: In get_port, handle "invalid" rates

where sysfs rate file contains "Invalid argument"

---

index 45a9423..c638ebd 100644
@@ -132,6 +132,7 @@ static int get_port(char *ca_name, char *dir, int portnum, umad_port_t * port)
 	uint8_t gid[16];
 	struct dirent **namelist = NULL;
 	int i, len, num_pkeys = 0;
+	char tmp[24];
 
 	strncpy(port->ca_name, ca_name, sizeof port->ca_name - 1);
 	port->portnum = portnum;

@@ -153,8 +154,13 @@ static int get_port(char *ca_name, char *dir, int portnum, umad_port_t * port)
 		goto clean;
 	if (sys_read_uint(port_dir, SYS_PORT_PHY_STATE, &port->phys_state) < 0)
 		goto clean;
-	if (sys_read_uint(port_dir, SYS_PORT_RATE, &port->rate) < 0)
-		goto clean;
+	if (sys_read_uint(port_dir, SYS_PORT_RATE, &port->rate) < 0) {
+		if (sys_read_string(port_dir, SYS_PORT_RATE, tmp,
+				    sizeof(tmp)) < 0)
+			goto clean;
+		if (strcmp(tmp, strerror(EINVAL)))
+			goto clean;
+	}
 	if (sys_read_uint(port_dir, SYS_PORT_CAPMASK, &port->capmask) < 0)
 		goto clean;
 

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Just to sync, I just sent kernel patches which should eliminate the case when EINVAL is seen by rate_show after calling into the mlx4 driver. This addresses point #2, as for #3, I made a check and forced Ethernet link layer on IB port, in this case the mlx4 driver issues the ethernet port query command and it returns valid value for active_speed (4) so I didn't patch that code too. As for libibumad fixes, I'm not dealing with that...

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

I agree they are not "RoCE ready". But the main reason is I am unclear what "RoCE ready" means. My first thought is that "InfiniBand" Diags should not function on an Ethernet link. However, we seem to be merging much of the functionality and it does not seem to hurt in most cases.

If some of the diags do retain functionality on an Ethernet link then perhaps some name changes are in order in addition to testing. For example "ibstat" should probably be "rdmastat" or something. (This change was made to the perftest package a long time ago.)

I guess my question to the hardware vendors is:

What MAD's, __if__ any, do you see Ethernet supporting in the future? Do you see MADs being used in some Open Flow spec to be able to program switches? What about Performance Management?

I don't want to get all draconian and remove these devices, since having more information (ie from ibstat) is good. But other than that tool what else should the diags support?

Ira

Ira Weiny
Member of Technical Staff
Lawrence Livermore National Lab
925-423-8008
[email protected]
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Some additional info:
- This issue only occurs if the back-to-back connected system is down, not if it is running.
- The output I get with the other system down is:

# cat /sys/class/infiniband/mlx4_0/ports/1/link_layer
InfiniBand
# dmesg -c >/dev/null
# cat /sys/class/infiniband/mlx4_0/ports/1/rate
cat: /sys/class/infiniband/mlx4_0/ports/1/rate: Invalid argument
# dmesg -c
ib_link_query_port active_speed 7
rate_show ret 0 for ib_query_port dev mlx4_0 port 1 link 1

Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

So you're getting the same wrong value of seven which I get on my systems, I think the patch / way to go here would be to assume some fixed speed (SDR?) when the link is down - as when the link is really down or down since the "wrong" link layer is assumed, the firmware command returns that value of seven which isn't in the speed enum, Roland?

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html