Intermitted "Failed to obtain HW semaphore, aborting" error

Intermittently a couple of nodes in our cluster throw the error "Failed to obtain HW semaphore, aborting" on boot. When this error occurs we are unable to use IB on those nodes, unloading and reloading the module doesnt help. I was wondering what could be causing this error, google only brings up the source code and no discussion about this error. We are using OFED-1.5.4, any help in debugging and resolving this issue is greatly appreciated.

note: I am not subscribed to the list so would appreciate if I am copied in the replies.

Bharath

load mlx4_core with debug_level=1 and send the resulted dmesg along with the lspci info of the card ("$ lspci | grep Mellanox")

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

The same node will come up fine on some reboots and on others I will get this error. Here is the output from lspci

$ lspci | grep Mellanox
01:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3]

dmesg output trying to load mlx4_core with debug_level=1

mlx4_core: Mellanox ConnectX core driver v1.0-ofed1.5.4 (November 10, 2011)
mlx4_core: Initializing 0000:01:00.0
mlx4_core 0000:01:00.0: PCI INT A -> GSI 26 (level, low) -> IRQ 26 mlx4_core 0000:01:00.0: setting latency timer to 64
Uhhuh. NMI received for unknown reason 31 on CPU 0.
Do you have a strange power saving mode enabled?
Dazed and confused, but trying to continue
Uhhuh. NMI received for unknown reason 31 on CPU 0.
Do you have a strange power saving mode enabled?
Dazed and confused, but trying to continue
Uhhuh. NMI received for unknown reason 21 on CPU 0.
Do you have a strange power saving mode enabled?
Dazed and confused, but trying to continue
mlx4_core 0000:01:00.0: Failed to obtain HW semaphore, aborting mlx4_core 0000:01:00.0: Failed to reset HCA, aborting. mlx4_core 0000:01:00.0: PCI INT A disabled
mlx4_core: probe of 0000:01:00.0 failed with error -11

I am unable to run ibv_devinfo on the bad node, here is an output from a good node

$ ibv_devinfo
hca_id: mlx4_0
        transport:                      InfiniBand (0)
        fw_ver:                         2.10.2370
        node_guid:                      001e:6703:003c:dff4
        sys_image_guid:                 001e:6703:003c:dff7
        vendor_id:                      0x02c9
        vendor_part_id:                 4099
        hw_ver:                         0x0
        board_id:                       INCX-3I358C10501
        phys_port_cnt:                  1
                port:   1
                        state:                  PORT_ACTIVE (4)
                        max_mtu:                2048 (4)
                        active_mtu:             2048 (4)
                        sm_lid:                 358
                        port_lid:               331
                        port_lmc:               0x00
                        link_layer:             IB

Bharath

On Sun, Feb 3, 2013 at 5:18 PM, Bharath Ramesh <[email protected]> wrote:
[...]

[...]

should be step in the right direction if you upgrade your firmware to the latest GA, 2.11.0500 avail on the Mellanox site.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html