IB / MLX setup issues

Also, and I'm unsure if this is related, the link between machines C and A is in the initializing state when I bring up the machine. Machine A has the subnet manager running, so I would think that should be enough. If I run another subnet manager on machine C, I get an active link, but I wouldn't think I need to be running two subnet managers.

Hi,

I'm running into some issues when trying to set up a new IB cluster. I've got machine A which has connections to machine B and C via a single two-port ConnectX card. A and B are connected and have no known problems. I just set up machine C (installed OFED-1.5.1) but whenever I run ibv_devinfo I get:

libibverbs: Warning: couldn't load driver 'mlx4':
/usr/local/lib/libmlx4-rdmav2.so: symbol ibv_cmd_get_eth_l2_addr, version IBVERBS_1.0 not defined in file libibverbs.so.1 with link time reference
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
No IB devices found

The machine does have 3 HCAs. For reference OFED 1.5.1 has deposited the following RPMs on the system:
libmlx4-1.0-0.7.g2432360 libmlx4-devel-1.0-0.7.g2432360 libibverbs-1.1.3-0.6.g932f1a2 libibverbs-devel-1.1.3-0.6.g932f1a2 libibverbs-devel-static-1.1.3-0.6.g932f1a2

Quoted 20 lines · Todd Strader

Hi,

I'm running into some issues when trying to set up a new IB cluster. I've got machine A which has connections to machine B and C via a single two-port ConnectX card. A and B are connected and have no known problems. I just set up machine C (installed OFED-1.5.1) but whenever I run ibv_devinfo I get:

libibverbs: Warning: couldn't load driver 'mlx4':
/usr/local/lib/libmlx4-rdmav2.so: symbol ibv_cmd_get_eth_l2_addr, version IBVERBS_1.0 not defined in file libibverbs.so.1 with link time reference
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
No IB devices found

Quoted 9 lines

The machine does have 3 HCAs. For reference OFED 1.5.1 has deposited the following RPMs on the system:
libmlx4-1.0-0.7.g2432360 libmlx4-devel-1.0-0.7.g2432360 libibverbs-1.1.3-0.6.g932f1a2 libibverbs-devel-1.1.3-0.6.g932f1a2 libibverbs-devel-static-1.1.3-0.6.g932f1a2

So you only have some installation problem on machine C. By default, OFED installation script will install to /usr and I see that in C the prefix is /usr/local - is this what you intended? I think you may have old libraries in the library path so you may need to remve them.