Skip to contentSkip to conversation

mlx4: propagate node_description changes down to FW to generate trap 144

PATCH In review Jack Morgenstein 2 participants 2 messages Oct 4, 2010 – Oct 23, 2010 1 message not in the archive

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 mlx4: propagate node_description changes down to FW to generate trap 144

The Node Description cannot be changed via MADs (it is read-only). Until now, it was changed in the driver via sysfs, and the new Node Description was simply inserted by the driver into MAD responses (replacing the description returned by FW).

Until now, openibd used the sysfs interface to change the node description at driver startup. However, that generated a race condition, where OpenSM could get the FW node description rather than the sysfs description if OpenSM queried the device before openibd had a chance to enter the new description.

The solution is a new FW command (SET_NODE) which allows passing the new node description to FW. When this command is invoked, FW issues a 144 trap to OpenSM. Upon receiving this trap, OpenSM can query the node to obtain the new node description -- thus eliminating the effects of the race.

This patch simply adds invoking the SET_NODE command when a new node description is entered via sysfs (thus causing trap 144 to be issued by the FW).

The patch works whether or not the new FW command is available. If SET_NODE is not available, things operate as before (i.e., the node description is changed in the driver, but no trap is issued).

---
Roland,

I left the snooping and overwriting the node description as is. That way, I do not need to test and remember if SET_NODE succeeded, and/or I do not need to know which FW version introduced SET_NODE.

Since we need to keep the old method (snooping and overwriting) around anyway, I think it is easier and simpler to keep things as they are, and only use SET_NODE to generate trap 144.

Please note that I do not generate any errors if SET_NODE fails for any reason, (including failure to allocate a mailbox) since the node description has been successfully entered into the driver and is available.

Jack

drivers/infiniband/hw/mlx4/main.c +22 -5
Index: infiniband/drivers/infiniband/hw/mlx4/main.c
===================================================================
@@ -272,14 +272,31 @@ out:
 static int mlx4_ib_modify_device(struct ib_device *ibdev, int mask,
 				 struct ib_device_modify *props)
 {
+	struct mlx4_cmd_mailbox *mailbox;
+
 	if (mask & ~IB_DEVICE_MODIFY_NODE_DESC)
 		return -EOPNOTSUPP;
 
-	if (mask & IB_DEVICE_MODIFY_NODE_DESC) {
-		spin_lock(&to_mdev(ibdev)->sm_lock);
-		memcpy(ibdev->node_desc, props->node_desc, 64);
-		spin_unlock(&to_mdev(ibdev)->sm_lock);
-	}
+	if (!(mask & IB_DEVICE_MODIFY_NODE_DESC))
+		return 0;
+
+	spin_lock(&to_mdev(ibdev)->sm_lock);
+	memcpy(ibdev->node_desc, props->node_desc, 64);
+	spin_unlock(&to_mdev(ibdev)->sm_lock);
+
+	/* if possible, pass node desc to FW, so it can generate
+	 * a 144 trap. If cmd fails, just ignore.
+	 */
+	mailbox = mlx4_alloc_cmd_mailbox(to_mdev(ibdev)->dev);
+	if (IS_ERR(mailbox))
+		return 0;
+
+	memset(mailbox->buf, 0, 256);
+	memcpy(mailbox->buf, props->node_desc, 64);
+	mlx4_cmd(to_mdev(ibdev)->dev, mailbox->dma, 1, 0,
+		 MLX4_CMD_SET_NODE, MLX4_CMD_TIME_CLASS_A);
+
+	mlx4_free_cmd_mailbox(to_mdev(ibdev)->dev, mailbox);
 
 	return 0;
 }
include/linux/mlx4/cmd.h +1 -0
Index: infiniband/include/linux/mlx4/cmd.h
===================================================================
@@ -57,6 +57,7 @@ enum {
 	MLX4_CMD_QUERY_PORT	 = 0x43,
 	MLX4_CMD_SENSE_PORT	 = 0x4d,
 	MLX4_CMD_SET_PORT	 = 0xc,
+	MLX4_CMD_SET_NODE	 = 0x5a,
 	MLX4_CMD_ACCESS_DDR	 = 0x2e,
 	MLX4_CMD_MAP_ICM	 = 0xffa,
 	MLX4_CMD_UNMAP_ICM	 = 0xff9,

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 201010041411.34956.jackm-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org