Skip to contentSkip to conversation

DEVX VHCA tunnel support

PATCH Applied Leon Romanovsky 2 participants 6 messages Jul 1, 2019 – Jul 8, 2019

Subsystems (5)
lore
Share & tools
lore
#0 lore raw

cover DEVX VHCA tunnel support

From: Leon Romanovsky <[email protected]>

Hi,

Those two patches introduce VHCA tunnel mechanism to DEVX interface needed for Bluefield SOC. See extensive commit messages for more information.

Thanks

Max Gurtovoy (2):
net/mlx5: Introduce VHCA tunnel device capability IB/mlx5: Implement VHCA tunnel mechanism in DEVX

 drivers/infiniband/hw/mlx5/devx.c | 24 ++++++++++++++++++++----
 include/linux/mlx5/mlx5_ifc.h     | 10 ++++++++--
 2 files changed, 28 insertions(+), 6 deletions(-)
#1 lore raw

PATCH 1/2 net/mlx5: Introduce VHCA tunnel device capability

From: Max Gurtovoy <[email protected]>

When using the device emulation feature (introduced in Bluefield-1 SOC), a privileged function (the device emulation manager) will be able to create a channel to execute commands on behalf of the emulated function.

This channel will be a general object of type VHCA_TUNNEL that will have a unique ID for each emulated function. This ID will be passed in each cmd that will be issued by the emulation SW in a well known offset in the command header.

This channel is needed since the emulated function doesn't have a normal command interface to the HCA HW, but some basic configuration for that function is needed (e.g. initialize and enable the HCA). For that matter, a specific command-set was defined and only those commands will be issued by the HCA.

---
 include/linux/mlx5/mlx5_ifc.h | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)
include/linux/mlx5/mlx5_ifc.h +8 -2
index 5d2bf91e130b..a35e545a8de0 100644
@@ -1329,7 +1329,13 @@ struct mlx5_ifc_cmd_hca_cap_bits {
 
 	u8         reserved_at_6c0[0x4];
 	u8         flex_parser_id_geneve_tlv_option_0[0x4];
-	u8         reserved_at_6c8[0x138];
+	u8         reserved_at_6c8[0x18];
+
+	u8         reserved_at_6e0[0xa0];
+
+	u8         vhca_tunnel_commands[0x40];
+
+	u8         reserved_at_7c0[0x40];
 };
 
 enum mlx5_flow_destination_type {
@@ -9560,7 +9566,7 @@ struct mlx5_ifc_general_obj_in_cmd_hdr_bits {
 	u8         opcode[0x10];
 	u8         uid[0x10];
 
-	u8         reserved_at_20[0x10];
+	u8         vhca_tunnel_id[0x10];
 	u8         obj_type[0x10];
 
 	u8         obj_id[0x20];
Signature · 1 line
2.20.1
#2 lore raw

PATCH 2/2 IB/mlx5: Implement VHCA tunnel mechanism in DEVX

In reply to Leon Romanovsky

From: Max Gurtovoy <[email protected]>

This mechanism will allow function-A to perform operations "on behalf" of function-B via tunnel object. Function-A will have privileges for creating and using this tunnel object.

For example, in the device emulation feature presented in Bluefield-1 SoC, using device emulation capability, one can present NVMe function to the host OS.

Since the NVMe function doesn't have a normal command interface to the HCA HW, here is a need to create a channel that will be able to issue commands "on behalf" of this function.

This channel is the VHCA_TUNNEL general object. The emulation software will create this tunnel for every managed function and issue commands via devx general cmd interface using the appropriate tunnel ID. When devX context will receive a command with non-zero vhca_tunnel_id, it will pass the command as-is down to the HCA.

All the validation, security and resource tracking of the commands and the created tunneled objects is in the responsibility of the HCA FW. When a VHCA_TUNNEL object destroyed, the device will issue an internal FLR (function level reset) to the emulated function associated with this tunnel. This will destroy all the created resources using the tunnel mechanism.

---
 drivers/infiniband/hw/mlx5/devx.c | 24 ++++++++++++++++++++----
 1 file changed, 20 insertions(+), 4 deletions(-)
drivers/infiniband/hw/mlx5/devx.c +20 -4
index cb4f0fc79176..26ba6bf5d19a 100644
@@ -894,12 +894,16 @@ static int devx_get_uid(struct mlx5_ib_ucontext *c, void *cmd_in)
 
 	return c->devx_uid;
 }
-static bool devx_is_general_cmd(void *in)
+
+static bool devx_is_general_cmd(void *in, struct mlx5_ib_dev *dev)
 {
 	u16 opcode = MLX5_GET(general_obj_in_cmd_hdr, in, opcode);
 
-	if (opcode >= MLX5_CMD_OP_GENERAL_START &&
-	    opcode < MLX5_CMD_OP_GENERAL_END)
+	/* Pass all cmds for vhca_tunnel as general, tracking is done in FW */
+	if ((MLX5_CAP_GEN_64(dev->mdev, vhca_tunnel_commands) &&
+	     MLX5_GET(general_obj_in_cmd_hdr, in, vhca_tunnel_id)) ||
+	    (opcode >= MLX5_CMD_OP_GENERAL_START &&
+	     opcode < MLX5_CMD_OP_GENERAL_END))
 		return true;
 
 	switch (opcode) {
@@ -1025,7 +1029,7 @@ static int UVERBS_HANDLER(MLX5_IB_METHOD_DEVX_OTHER)(
 		return uid;
 
 	/* Only white list of some general HCA commands are allowed for this method. */
-	if (!devx_is_general_cmd(cmd_in))
+	if (!devx_is_general_cmd(cmd_in, dev))
 		return -EINVAL;
 
 	cmd_out = uverbs_zalloc(attrs, cmd_out_len);
@@ -1420,6 +1424,9 @@ static int UVERBS_HANDLER(MLX5_IB_METHOD_DEVX_OBJ_CREATE)(
 	u32 obj_id;
 	u16 opcode;
 
+	if (MLX5_GET(general_obj_in_cmd_hdr, cmd_in, vhca_tunnel_id))
+		return -EINVAL;
+
 	uid = devx_get_uid(c, cmd_in);
 	if (uid < 0)
 		return uid;
@@ -1519,6 +1526,9 @@ static int UVERBS_HANDLER(MLX5_IB_METHOD_DEVX_OBJ_MODIFY)(
 	int err;
 	int uid;
 
+	if (MLX5_GET(general_obj_in_cmd_hdr, cmd_in, vhca_tunnel_id))
+		return -EINVAL;
+
 	uid = devx_get_uid(c, cmd_in);
 	if (uid < 0)
 		return uid;
@@ -1561,6 +1571,9 @@ static int UVERBS_HANDLER(MLX5_IB_METHOD_DEVX_OBJ_QUERY)(
 	int uid;
 	struct mlx5_ib_dev *mdev = to_mdev(c->ibucontext.device);
 
+	if (MLX5_GET(general_obj_in_cmd_hdr, cmd_in, vhca_tunnel_id))
+		return -EINVAL;
+
 	uid = devx_get_uid(c, cmd_in);
 	if (uid < 0)
 		return uid;
@@ -1698,6 +1711,9 @@ static int UVERBS_HANDLER(MLX5_IB_METHOD_DEVX_OBJ_ASYNC_QUERY)(
 	struct devx_async_cmd_event_file *ev_file;
 	struct devx_async_data *async_data;
 
+	if (MLX5_GET(general_obj_in_cmd_hdr, cmd_in, vhca_tunnel_id))
+		return -EINVAL;
+
 	uid = devx_get_uid(c, cmd_in);
 	if (uid < 0)
 		return uid;
Signature · 1 line
2.20.1
#3 lore raw
In reply to Leon Romanovsky
Quoted 10 lines · Leon Romanovsky

From: Leon Romanovsky <[email protected]>

Hi,

Those two patches introduce VHCA tunnel mechanism to DEVX interface needed for Bluefield SOC. See extensive commit messages for more information.

Thanks

Quoted 7 lines

Max Gurtovoy (2):
net/mlx5: Introduce VHCA tunnel device capability IB/mlx5: Implement VHCA tunnel mechanism in DEVX

 drivers/infiniband/hw/mlx5/devx.c | 24 ++++++++++++++++++++----
 include/linux/mlx5/mlx5_ifc.h     | 10 ++++++++--
 2 files changed, 28 insertions(+), 6 deletions(-)

This looks Ok can you apply the mlx5-next patch please

Thanks,
Jason

#4 lore raw
Quoted 15 lines · Jason Gunthorpe
Quoted 14 lines · Leon Romanovsky

From: Leon Romanovsky <[email protected]>

Hi,

Those two patches introduce VHCA tunnel mechanism to DEVX interface needed for Bluefield SOC. See extensive commit messages for more information.

Thanks

Max Gurtovoy (2):
net/mlx5: Introduce VHCA tunnel device capability IB/mlx5: Implement VHCA tunnel mechanism in DEVX

Quoted 5 lines
Quoted 3 lines
 drivers/infiniband/hw/mlx5/devx.c | 24 ++++++++++++++++++++----
 include/linux/mlx5/mlx5_ifc.h     | 10 ++++++++--
 2 files changed, 28 insertions(+), 6 deletions(-)

This looks Ok can you apply the mlx5-next patch please

1dd7382b1bb6 net/mlx5: Introduce VHCA tunnel device capability

Thanks

Quoted 4 lines

Thanks,
Jason

#5 lore raw
In reply to Leon Romanovsky
Quoted 8 lines · Leon Romanovsky

From: Leon Romanovsky <[email protected]>

Hi,

Those two patches introduce VHCA tunnel mechanism to DEVX interface needed for Bluefield SOC. See extensive commit messages for more information.

Quoted 5 lines

Thanks

Max Gurtovoy (2):
net/mlx5: Introduce VHCA tunnel device capability IB/mlx5: Implement VHCA tunnel mechanism in DEVX

Thanks, applied to for-next

Jason