Skip to contentSkip to conversation

mlx5 updates 2023-06-09

GIT PULL Applied Saeed Mahameed 6 participants 41 messages Jun 10, 2023 – Jun 27, 2023

Subsystems (5)
lore
Share & tools
lore
#0 lore raw

From: Saeed Mahameed <[email protected]>

This pull adds the following to mlx5 features:
1) Embedded CPU Virtual Functions
2) Lightweight local SFs

For more information please see tag log below.

Please pull and let me know if there is any problem.

Thanks,
Saeed.

The following changes since commit ded5c1a16ec69bb815f2b7d9ea4028913ebffca4:

Merge branch 'tools-ynl-gen-code-gen-improvements-before-ethtool' (2023-06-09 14:40:33 -0700)

are available in the Git repository at:

git://git.kernel.org/pub/scm/linux/kernel/git/saeed/linux.git tags/mlx5-updates-2023-06-09

for you to fetch changes up to 978015f7ef9240acfb078f4c1c0d79459b42f951:

net/mlx5e: Remove a useless function call (2023-06-09 18:40:53 -0700)

----------------------------------------------------------------
mlx5-updates-2023-06-09

1) Embedded CPU Virtual Functions
2) Lightweight local SFs

Daniel Jurgens says:
====================
Embedded CPU Virtual Functions

This series enables the creation of virtual functions on Bluefield (the embedded CPU platform). Embedded CPU virtual functions (EC VFs). EC VF creation, deletion and management interfaces are the same as those for virtual functions in a server with a Connect-X NIC.

When using EC VFs on the ARM the creation of virtual functions on the host system is still supported. Host VFs eswitch vports occupy a range of 1..max_vfs, the EC VF vport range is max_vfs+1..max_ec_vfs.

Every function (PF, ECPF, VF, EC VF, and subfunction) has a function ID associated with it. Prior to this series the function ID and the eswitch vport were the same. That is no longer the case, the EC VF function ID range is 1..max_ec_vfs. When querying or setting the capabilities of an EC VF function an new bit must be set in the query/set HCA cap structure.

This is a high level overview of the changes made:
- Allocate vports for EC VFs if they are enabled.
- Create representors and devlink ports for the EC VF vports.
- When querying/setting HCA caps by vport break the assumption that function ID is the same a vport number and adjust accordingly.
- Create a new type of page, so that when SRIOV on the ARM is disabled, but remains enabled on the host, the driver can wait for the correct pages.
- Update SRIOV code to support EC VF creation/deletion.

===================

Lightweight local SFs:

Last 3 patches form Shay Drory:

SFs are heavy weight and by default they come with the full package of ConnectX features. Usually users want specialized SFs for one specific purpose and using devlink users will almost always override the set of advertises features of an SF and reload it.

Shay Drory says:
================
In order to avoid the wasted time and resources on the reload, local SFs will probe without any auxiliary sub-device, so that the SFs can be configured prior to its full probe.

The defaults of the enable_* devlink params of these SFs are set to false.

Usage example:
Create SF:

$ devlink port add pci/0000:08:00.0 flavour pcisf pfnum 0 sfnum 11
$ devlink port function set pci/0000:08:00.0/32768 \
               hw_addr 00:00:00:00:00:11 state active

Enable ETH auxiliary device:

$ devlink dev param set auxiliary/mlx5_core.sf.1 \
              name enable_eth value true cmode driverinit

Now, in order to fully probe the SF, use devlink reload:

$ devlink dev reload auxiliary/mlx5_core.sf.1

At this point the user have SF devlink instance with auxiliary device for the Ethernet functionality only.

================

----------------------------------------------------------------
Christophe JAILLET (1):
net/mlx5e: Remove a useless function call

Daniel Jurgens (11):
net/mlx5: Simplify unload all rep code net/mlx5: mlx5_ifc updates for embedded CPU SRIOV net/mlx5: Enable devlink port for embedded cpu VF vports net/mlx5: Update vport caps query/set for EC VFs net/mlx5: Add management of EC VF vports net/mlx5: Add/remove peer miss rules for EC VFs net/mlx5: Add new page type for EC VF pages net/mlx5: Use correct vport when restoring GUIDs net/mlx5: Query correct caps for min msix vectors net/mlx5: Update SRIOV enable/disable to handle EC/VFs net/mlx5: Set max number of embedded CPU VFs

Shay Drory (3):
net/mlx5: Split function_setup() to enable and open functions net/mlx5: Move esw multiport devlink param to eswitch code net/mlx5: Light probe local SFs

 .../ethernet/mellanox/mlx5/switchdev.rst           |  20 ++
 drivers/net/ethernet/mellanox/mlx5/core/debugfs.c  |   1 +
 drivers/net/ethernet/mellanox/mlx5/core/dev.c      |  16 ++
 drivers/net/ethernet/mellanox/mlx5/core/devlink.c  |  54 ++----
 .../ethernet/mellanox/mlx5/core/en/tc/post_act.c   |   4 +-
 .../ethernet/mellanox/mlx5/core/esw/devlink_port.c |   8 +-
 drivers/net/ethernet/mellanox/mlx5/core/eswitch.c  | 174 +++++++++++++++--
 drivers/net/ethernet/mellanox/mlx5/core/eswitch.h  |  13 ++
 .../ethernet/mellanox/mlx5/core/eswitch_offloads.c | 102 +++++-----
 drivers/net/ethernet/mellanox/mlx5/core/health.c   |  24 ++-
 drivers/net/ethernet/mellanox/mlx5/core/main.c     | 209 +++++++++++++++++----
 .../net/ethernet/mellanox/mlx5/core/mlx5_core.h    |  35 +++-
 .../net/ethernet/mellanox/mlx5/core/pagealloc.c    |  11 +-
 drivers/net/ethernet/mellanox/mlx5/core/pci_irq.c  |  16 +-
 .../ethernet/mellanox/mlx5/core/sf/dev/driver.c    |  15 +-
 drivers/net/ethernet/mellanox/mlx5/core/sriov.c    |  46 ++++-
 drivers/net/ethernet/mellanox/mlx5/core/vport.c    |  19 +-
 include/linux/mlx5/driver.h                        |   7 +
 include/linux/mlx5/mlx5_ifc.h                      |  11 +-
 include/linux/mlx5/vport.h                         |   2 +-
 20 files changed, 610 insertions(+), 177 deletions(-)
#1 lore raw

PATCH 1/15 net/mlx5: Simplify unload all rep code

From: Daniel Jurgens <[email protected]>

Instead of using type specific iterators which are only used in one place just traverse the xarray. It will provide suitable ordering based on the vport numbers. This will also eliminate the need for changes here when new types are added.

---
 .../mellanox/mlx5/core/eswitch_offloads.c     | 48 +------------------
 1 file changed, 1 insertion(+), 47 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +1 -47
index eafb098db6b0..625982454575 100644
@@ -55,13 +55,6 @@
 #define mlx5_esw_for_each_rep(esw, i, rep) \
 	xa_for_each(&((esw)->offloads.vport_reps), i, rep)
 
-#define mlx5_esw_for_each_sf_rep(esw, i, rep) \
-	xa_for_each_marked(&((esw)->offloads.vport_reps), i, rep, MLX5_ESW_VPT_SF)
-
-#define mlx5_esw_for_each_vf_rep(esw, index, rep)	\
-	mlx5_esw_for_each_entry_marked(&((esw)->offloads.vport_reps), index, \
-				       rep, (esw)->esw_funcs.num_vfs, MLX5_ESW_VPT_VF)
-
 /* There are two match-all miss flows, one for unicast dst mac and
  * one for multicast.
  */
@@ -2191,18 +2184,6 @@ static int esw_offloads_start(struct mlx5_eswitch *esw,
 	return 0;
 }
 
-static void mlx5_esw_offloads_rep_mark_set(struct mlx5_eswitch *esw,
-					   struct mlx5_eswitch_rep *rep,
-					   xa_mark_t mark)
-{
-	bool mark_set;
-
-	/* Copy the mark from vport to its rep */
-	mark_set = xa_get_mark(&esw->vports, rep->vport, mark);
-	if (mark_set)
-		xa_set_mark(&esw->offloads.vport_reps, rep->vport, mark);
-}
-
 static int mlx5_esw_offloads_rep_init(struct mlx5_eswitch *esw, const struct mlx5_vport *vport)
 {
 	struct mlx5_eswitch_rep *rep;
@@ -2222,9 +2203,6 @@ static int mlx5_esw_offloads_rep_init(struct mlx5_eswitch *esw, const struct mlx
 	if (err)
 		goto insert_err;
 
-	mlx5_esw_offloads_rep_mark_set(esw, rep, MLX5_ESW_VPT_HOST_FN);
-	mlx5_esw_offloads_rep_mark_set(esw, rep, MLX5_ESW_VPT_VF);
-	mlx5_esw_offloads_rep_mark_set(esw, rep, MLX5_ESW_VPT_SF);
 	return 0;
 
 insert_err:
@@ -2365,37 +2343,13 @@ static void __esw_offloads_unload_rep(struct mlx5_eswitch *esw,
 		esw->offloads.rep_ops[rep_type]->unload(rep);
 }
 
-static void __unload_reps_sf_vport(struct mlx5_eswitch *esw, u8 rep_type)
-{
-	struct mlx5_eswitch_rep *rep;
-	unsigned long i;
-
-	mlx5_esw_for_each_sf_rep(esw, i, rep)
-		__esw_offloads_unload_rep(esw, rep, rep_type);
-}
-
 static void __unload_reps_all_vport(struct mlx5_eswitch *esw, u8 rep_type)
 {
 	struct mlx5_eswitch_rep *rep;
 	unsigned long i;
 
-	__unload_reps_sf_vport(esw, rep_type);
-
-	mlx5_esw_for_each_vf_rep(esw, i, rep)
-		__esw_offloads_unload_rep(esw, rep, rep_type);
-
-	if (mlx5_ecpf_vport_exists(esw->dev)) {
-		rep = mlx5_eswitch_get_rep(esw, MLX5_VPORT_ECPF);
-		__esw_offloads_unload_rep(esw, rep, rep_type);
-	}
-
-	if (mlx5_core_is_ecpf_esw_manager(esw->dev)) {
-		rep = mlx5_eswitch_get_rep(esw, MLX5_VPORT_PF);
+	mlx5_esw_for_each_rep(esw, i, rep)
 		__esw_offloads_unload_rep(esw, rep, rep_type);
-	}
-
-	rep = mlx5_eswitch_get_rep(esw, MLX5_VPORT_UPLINK);
-	__esw_offloads_unload_rep(esw, rep, rep_type);
 }
 
 int mlx5_esw_offloads_rep_load(struct mlx5_eswitch *esw, u16 vport_num)
Signature · 1 line
2.40.1
#2 lore raw

Re: [net-next 01/15] net/mlx5: Simplify unload all rep code

This series was applied to netdev/net-next.git (main) by Saeed Mahameed <[email protected]>:

Hello:

This series was applied to netdev/net-next.git (main) by Saeed Mahameed <[email protected]>:

Quoted 8 lines · On Fri, 9 Jun 2023 18:42:40 -0700 you wrote:

From: Daniel Jurgens <[email protected]>

Instead of using type specific iterators which are only used in one place just traverse the xarray. It will provide suitable ordering based on the vport numbers. This will also eliminate the need for changes here when new types are added.

[...]

Here is the summary with links:
- [net-next,01/15] net/mlx5: Simplify unload all rep code

    https://git.kernel.org/netdev/net-next/c/18a92b054254
  - [net-next,02/15] net/mlx5: mlx5_ifc updates for embedded CPU SRIOV
    https://git.kernel.org/netdev/net-next/c/93b36d0f2892
  - [net-next,03/15] net/mlx5: Enable devlink port for embedded cpu VF vports
    https://git.kernel.org/netdev/net-next/c/dc13180824b7
  - [net-next,04/15] net/mlx5: Update vport caps query/set for EC VFs
    https://git.kernel.org/netdev/net-next/c/9ac0b128248e
  - [net-next,05/15] net/mlx5: Add management of EC VF vports
    https://git.kernel.org/netdev/net-next/c/a7719b29a821
  - [net-next,06/15] net/mlx5: Add/remove peer miss rules for EC VFs
    https://git.kernel.org/netdev/net-next/c/fa3c73eee641
  - [net-next,07/15] net/mlx5: Add new page type for EC VF pages
    https://git.kernel.org/netdev/net-next/c/395ccd6eb49a
  - [net-next,08/15] net/mlx5: Use correct vport when restoring GUIDs
    https://git.kernel.org/netdev/net-next/c/2ee3db806e85
  - [net-next,09/15] net/mlx5: Query correct caps for min msix vectors
    https://git.kernel.org/netdev/net-next/c/42a84a430931
  - [net-next,10/15] net/mlx5: Update SRIOV enable/disable to handle EC/VFs
    https://git.kernel.org/netdev/net-next/c/6d98f314bfca
  - [net-next,11/15] net/mlx5: Set max number of embedded CPU VFs
    https://git.kernel.org/netdev/net-next/c/7057fe561988
  - [net-next,12/15] net/mlx5: Split function_setup() to enable and open functions
    https://git.kernel.org/netdev/net-next/c/2059cf51f318
  - [net-next,13/15] net/mlx5: Move esw multiport devlink param to eswitch code
    https://git.kernel.org/netdev/net-next/c/3f90840305e2
  - [net-next,14/15] net/mlx5: Light probe local SFs
    https://git.kernel.org/netdev/net-next/c/e71383fb9cd1
  - [net-next,15/15] net/mlx5e: Remove a useless function call
    https://git.kernel.org/netdev/net-next/c/978015f7ef92

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html
#3 lore raw

PATCH 2/15 net/mlx5: mlx5_ifc updates for embedded CPU SRIOV

In reply to Saeed Mahameed

From: Daniel Jurgens <[email protected]>

Add ec_vf_vport_base to HCA Capabilities 2. This indicates the base vport of embedded CPU virtual functions that are connected to the eswitch.

Add ec_vf_function to query/set_hca_caps. If set this indicates accessing a virtual function on the embedded CPU by function ID. This should only be used with other_function set to 1.

---
 include/linux/mlx5/mlx5_ifc.h | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)
include/linux/mlx5/mlx5_ifc.h +8 -3
index af3a92ad2e6b..1f4f62cb9f34 100644
@@ -1992,7 +1992,10 @@ struct mlx5_ifc_cmd_hca_cap_2_bits {
 	u8	   ts_cqe_metadata_size2wqe_counter[0x5];
 	u8	   reserved_at_250[0x10];
 
-	u8	   reserved_at_260[0x5a0];
+	u8	   reserved_at_260[0x120];
+	u8	   reserved_at_380[0x10];
+	u8	   ec_vf_vport_base[0x10];
+	u8	   reserved_at_3a0[0x460];
 };
 
 enum mlx5_ifc_flow_destination_type {
@@ -4805,7 +4808,8 @@ struct mlx5_ifc_set_hca_cap_in_bits {
 	u8         op_mod[0x10];
 
 	u8         other_function[0x1];
-	u8         reserved_at_41[0xf];
+	u8         ec_vf_function[0x1];
+	u8         reserved_at_42[0xe];
 	u8         function_id[0x10];
 
 	u8         reserved_at_60[0x20];
@@ -5956,7 +5960,8 @@ struct mlx5_ifc_query_hca_cap_in_bits {
 	u8         op_mod[0x10];
 
 	u8         other_function[0x1];
-	u8         reserved_at_41[0xf];
+	u8         ec_vf_function[0x1];
+	u8         reserved_at_42[0xe];
 	u8         function_id[0x10];
 
 	u8         reserved_at_60[0x20];
Signature · 1 line
2.40.1
#4 lore raw

PATCH 3/15 net/mlx5: Enable devlink port for embedded cpu VF vports

In reply to Saeed Mahameed

From: Daniel Jurgens <[email protected]>

Enable creation of a devlink port for EC VF vports.

---
 .../mellanox/mlx5/core/esw/devlink_port.c     |  8 +++++++-
 .../ethernet/mellanox/mlx5/core/mlx5_core.h   | 20 +++++++++++++++++++
 include/linux/mlx5/driver.h                   |  6 ++++++
 3 files changed, 33 insertions(+), 1 deletion(-)
drivers/net/ethernet/mellanox/mlx5/core/esw/devlink_port.c +7 -1
index f370f67d9e33..af779c700278 100644
@@ -18,7 +18,8 @@ static bool mlx5_esw_devlink_port_supported(struct mlx5_eswitch *esw, u16 vport_
 {
 	return vport_num == MLX5_VPORT_UPLINK ||
 	       (mlx5_core_is_ecpf(esw->dev) && vport_num == MLX5_VPORT_PF) ||
-	       mlx5_eswitch_is_vf_vport(esw, vport_num);
+	       mlx5_eswitch_is_vf_vport(esw, vport_num) ||
+	       mlx5_core_is_ec_vf_vport(esw->dev, vport_num);
 }
 
 static struct devlink_port *mlx5_esw_dl_port_alloc(struct mlx5_eswitch *esw, u16 vport_num)
@@ -56,6 +57,11 @@ static struct devlink_port *mlx5_esw_dl_port_alloc(struct mlx5_eswitch *esw, u16
 		dl_port->attrs.switch_id.id_len = ppid.id_len;
 		devlink_port_attrs_pci_vf_set(dl_port, controller_num, pfnum,
 					      vport_num - 1, external);
+	}  else if (mlx5_core_is_ec_vf_vport(esw->dev, vport_num)) {
+		memcpy(dl_port->attrs.switch_id.id, ppid.id, ppid.id_len);
+		dl_port->attrs.switch_id.id_len = ppid.id_len;
+		devlink_port_attrs_pci_vf_set(dl_port, controller_num, pfnum,
+					      vport_num - 1, false);
 	}
 	return dl_port;
 }
drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h +20 -0
index 1d879374acaa..0e7b5c6e4020 100644
@@ -343,4 +343,24 @@ bool mlx5_rdma_supported(struct mlx5_core_dev *dev);
 bool mlx5_vnet_supported(struct mlx5_core_dev *dev);
 bool mlx5_same_hw_devs(struct mlx5_core_dev *dev, struct mlx5_core_dev *peer_dev);
 
+static inline u16 mlx5_core_ec_vf_vport_base(const struct mlx5_core_dev *dev)
+{
+	return MLX5_CAP_GEN_2(dev, ec_vf_vport_base);
+}
+
+static inline u16 mlx5_core_ec_sriov_enabled(const struct mlx5_core_dev *dev)
+{
+	return mlx5_core_is_ecpf(dev) && mlx5_core_ec_vf_vport_base(dev);
+}
+
+static inline bool mlx5_core_is_ec_vf_vport(const struct mlx5_core_dev *dev, u16 vport_num)
+{
+	int base_vport = mlx5_core_ec_vf_vport_base(dev);
+	int max_vport = base_vport + mlx5_core_max_ec_vfs(dev);
+
+	if (!mlx5_core_ec_sriov_enabled(dev))
+		return false;
+
+	return (vport_num >= base_vport && vport_num < max_vport);
+}
 #endif /* __MLX5_CORE_H__ */
include/linux/mlx5/driver.h +6 -0
index 9a744c48eec2..252b6a6965b8 100644
@@ -474,6 +474,7 @@ struct mlx5_core_sriov {
 	struct mlx5_vf_context	*vfs_ctx;
 	int			num_vfs;
 	u16			max_vfs;
+	u16			max_ec_vfs;
 };
 
 struct mlx5_fc_pool {
@@ -1244,6 +1245,11 @@ static inline u16 mlx5_core_max_vfs(const struct mlx5_core_dev *dev)
 	return dev->priv.sriov.max_vfs;
 }
 
+static inline u16 mlx5_core_max_ec_vfs(const struct mlx5_core_dev *dev)
+{
+	return dev->priv.sriov.max_ec_vfs;
+}
+
 static inline int mlx5_get_gid_table_len(u16 param)
 {
 	if (param > 4) {
Signature · 1 line
2.40.1
#5 lore raw

PATCH 4/15 net/mlx5: Update vport caps query/set for EC VFs

In reply to Saeed Mahameed

From: Daniel Jurgens <[email protected]>

These functions are for query/set by vport, there was an underlying assumption that vport was equal to function ID. That's not the case for EC VF functions. Set the ec_vf_function bit accordingly.

---
 .../ethernet/mellanox/mlx5/core/mlx5_core.h   |  6 +++---
 .../net/ethernet/mellanox/mlx5/core/vport.c   | 19 +++++++++++++++----
 include/linux/mlx5/vport.h                    |  2 +-
 3 files changed, 19 insertions(+), 8 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h +3 -3
index 0e7b5c6e4020..7ca0c7a547aa 100644
@@ -325,10 +325,10 @@ void mlx5_unload_one_devl_locked(struct mlx5_core_dev *dev, bool suspend);
 int mlx5_load_one(struct mlx5_core_dev *dev, bool recovery);
 int mlx5_load_one_devl_locked(struct mlx5_core_dev *dev, bool recovery);
 
-int mlx5_vport_set_other_func_cap(struct mlx5_core_dev *dev, const void *hca_cap, u16 function_id,
+int mlx5_vport_set_other_func_cap(struct mlx5_core_dev *dev, const void *hca_cap, u16 vport,
 				  u16 opmod);
-#define mlx5_vport_get_other_func_general_cap(dev, fid, out)		\
-	mlx5_vport_get_other_func_cap(dev, fid, out, MLX5_CAP_GENERAL)
+#define mlx5_vport_get_other_func_general_cap(dev, vport, out)		\
+	mlx5_vport_get_other_func_cap(dev, vport, out, MLX5_CAP_GENERAL)
 
 void mlx5_events_work_enqueue(struct mlx5_core_dev *dev, struct work_struct *work);
 static inline u32 mlx5_sriov_get_vf_total_msix(struct pci_dev *pdev)
drivers/net/ethernet/mellanox/mlx5/core/vport.c +15 -4
index bc66b078a8a1..6d3984dd5b21 100644
@@ -1161,23 +1161,32 @@ u64 mlx5_query_nic_system_image_guid(struct mlx5_core_dev *mdev)
 }
 EXPORT_SYMBOL_GPL(mlx5_query_nic_system_image_guid);
 
-int mlx5_vport_get_other_func_cap(struct mlx5_core_dev *dev, u16 function_id, void *out,
+static int mlx5_vport_to_func_id(const struct mlx5_core_dev *dev, u16 vport, bool ec_vf_func)
+{
+	return ec_vf_func ? vport - mlx5_core_ec_vf_vport_base(dev)
+			  : vport;
+}
+
+int mlx5_vport_get_other_func_cap(struct mlx5_core_dev *dev, u16 vport, void *out,
 				  u16 opmod)
 {
+	bool ec_vf_func = mlx5_core_is_ec_vf_vport(dev, vport);
 	u8 in[MLX5_ST_SZ_BYTES(query_hca_cap_in)] = {};
 
 	opmod = (opmod << 1) | (HCA_CAP_OPMOD_GET_MAX & 0x01);
 	MLX5_SET(query_hca_cap_in, in, opcode, MLX5_CMD_OP_QUERY_HCA_CAP);
 	MLX5_SET(query_hca_cap_in, in, op_mod, opmod);
-	MLX5_SET(query_hca_cap_in, in, function_id, function_id);
+	MLX5_SET(query_hca_cap_in, in, function_id, mlx5_vport_to_func_id(dev, vport, ec_vf_func));
 	MLX5_SET(query_hca_cap_in, in, other_function, true);
+	MLX5_SET(query_hca_cap_in, in, ec_vf_function, ec_vf_func);
 	return mlx5_cmd_exec_inout(dev, query_hca_cap, in, out);
 }
 EXPORT_SYMBOL_GPL(mlx5_vport_get_other_func_cap);
 
 int mlx5_vport_set_other_func_cap(struct mlx5_core_dev *dev, const void *hca_cap,
-				  u16 function_id, u16 opmod)
+				  u16 vport, u16 opmod)
 {
+	bool ec_vf_func = mlx5_core_is_ec_vf_vport(dev, vport);
 	int set_sz = MLX5_ST_SZ_BYTES(set_hca_cap_in);
 	void *set_hca_cap;
 	void *set_ctx;
@@ -1191,8 +1200,10 @@ int mlx5_vport_set_other_func_cap(struct mlx5_core_dev *dev, const void *hca_cap
 	MLX5_SET(set_hca_cap_in, set_ctx, op_mod, opmod << 1);
 	set_hca_cap = MLX5_ADDR_OF(set_hca_cap_in, set_ctx, capability);
 	memcpy(set_hca_cap, hca_cap, MLX5_ST_SZ_BYTES(cmd_hca_cap));
-	MLX5_SET(set_hca_cap_in, set_ctx, function_id, function_id);
+	MLX5_SET(set_hca_cap_in, set_ctx, function_id,
+		 mlx5_vport_to_func_id(dev, vport, ec_vf_func));
 	MLX5_SET(set_hca_cap_in, set_ctx, other_function, true);
+	MLX5_SET(set_hca_cap_in, set_ctx, ec_vf_function, ec_vf_func);
 	ret = mlx5_cmd_exec_in(dev, set_hca_cap, set_ctx);
 
 	kfree(set_ctx);
include/linux/mlx5/vport.h +1 -1
index 7f31432f44c2..fbb9bf447889 100644
@@ -132,6 +132,6 @@ int mlx5_nic_vport_affiliate_multiport(struct mlx5_core_dev *master_mdev,
 int mlx5_nic_vport_unaffiliate_multiport(struct mlx5_core_dev *port_mdev);
 
 u64 mlx5_query_nic_system_image_guid(struct mlx5_core_dev *mdev);
-int mlx5_vport_get_other_func_cap(struct mlx5_core_dev *dev, u16 function_id, void *out,
+int mlx5_vport_get_other_func_cap(struct mlx5_core_dev *dev, u16 vport, void *out,
 				  u16 opmod);
 #endif /* __MLX5_VPORT_H__ */
Signature · 1 line
2.40.1
#6 lore raw

PATCH 5/15 net/mlx5: Add management of EC VF vports

In reply to Saeed Mahameed

From: Daniel Jurgens <[email protected]>

Add init, load, unload, and cleanup of the EC VF vports. This includes changes in how eswitch SRIOV is managed. Previous on an embedded CPU platform the number of VFs provided when enabling the eswitch was always 0, host VFs vports are handled in the eswitch functions change event handler. Now track the number of EC VFs as well, so they can be handled properly in the enable/disable flows.

There are only 3 marks available for use in xarrays, all 3 were already in use for this use case. EC VF vports are in a known range so we can access them by index instead of marks.

---
 .../net/ethernet/mellanox/mlx5/core/eswitch.c | 125 +++++++++++++++---
 .../net/ethernet/mellanox/mlx5/core/eswitch.h |  13 ++
 .../mellanox/mlx5/core/eswitch_offloads.c     |  22 +++
 3 files changed, 143 insertions(+), 17 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/eswitch.c +108 -17
index ecd8864d5d11..b33d852aae34 100644
@@ -1051,6 +1051,18 @@ static void mlx5_eswitch_clear_vf_vports_info(struct mlx5_eswitch *esw)
 	}
 }
 
+static void mlx5_eswitch_clear_ec_vf_vports_info(struct mlx5_eswitch *esw)
+{
+	struct mlx5_vport *vport;
+	unsigned long i;
+
+	mlx5_esw_for_each_ec_vf_vport(esw, i, vport, esw->esw_funcs.num_ec_vfs) {
+		memset(&vport->qos, 0, sizeof(vport->qos));
+		memset(&vport->info, 0, sizeof(vport->info));
+		vport->info.link_state = MLX5_VPORT_ADMIN_STATE_AUTO;
+	}
+}
+
 /* Public E-Switch API */
 int mlx5_eswitch_load_vport(struct mlx5_eswitch *esw, u16 vport_num,
 			    enum mlx5_eswitch_vport_event enabled_events)
@@ -1090,6 +1102,19 @@ void mlx5_eswitch_unload_vf_vports(struct mlx5_eswitch *esw, u16 num_vfs)
 	}
 }
 
+static void mlx5_eswitch_unload_ec_vf_vports(struct mlx5_eswitch *esw,
+					     u16 num_ec_vfs)
+{
+	struct mlx5_vport *vport;
+	unsigned long i;
+
+	mlx5_esw_for_each_ec_vf_vport(esw, i, vport, num_ec_vfs) {
+		if (!vport->enabled)
+			continue;
+		mlx5_eswitch_unload_vport(esw, vport->vport);
+	}
+}
+
 int mlx5_eswitch_load_vf_vports(struct mlx5_eswitch *esw, u16 num_vfs,
 				enum mlx5_eswitch_vport_event enabled_events)
 {
@@ -1110,6 +1135,26 @@ int mlx5_eswitch_load_vf_vports(struct mlx5_eswitch *esw, u16 num_vfs,
 	return err;
 }
 
+static int mlx5_eswitch_load_ec_vf_vports(struct mlx5_eswitch *esw, u16 num_ec_vfs,
+					  enum mlx5_eswitch_vport_event enabled_events)
+{
+	struct mlx5_vport *vport;
+	unsigned long i;
+	int err;
+
+	mlx5_esw_for_each_ec_vf_vport(esw, i, vport, num_ec_vfs) {
+		err = mlx5_eswitch_load_vport(esw, vport->vport, enabled_events);
+		if (err)
+			goto vf_err;
+	}
+
+	return 0;
+
+vf_err:
+	mlx5_eswitch_unload_ec_vf_vports(esw, num_ec_vfs);
+	return err;
+}
+
 static int host_pf_enable_hca(struct mlx5_core_dev *dev)
 {
 	if (!mlx5_core_is_ecpf(dev))
@@ -1154,6 +1199,12 @@ mlx5_eswitch_enable_pf_vf_vports(struct mlx5_eswitch *esw,
 		ret = mlx5_eswitch_load_vport(esw, MLX5_VPORT_ECPF, enabled_events);
 		if (ret)
 			goto ecpf_err;
+		if (mlx5_core_ec_sriov_enabled(esw->dev)) {
+			ret = mlx5_eswitch_load_ec_vf_vports(esw, esw->esw_funcs.num_ec_vfs,
+							     enabled_events);
+			if (ret)
+				goto ec_vf_err;
+		}
 	}
 
 	/* Enable VF vports */
@@ -1164,6 +1215,9 @@ mlx5_eswitch_enable_pf_vf_vports(struct mlx5_eswitch *esw,
 	return 0;
 
 vf_err:
+	if (mlx5_core_ec_sriov_enabled(esw->dev))
+		mlx5_eswitch_unload_ec_vf_vports(esw, esw->esw_funcs.num_ec_vfs);
+ec_vf_err:
 	if (mlx5_ecpf_vport_exists(esw->dev))
 		mlx5_eswitch_unload_vport(esw, MLX5_VPORT_ECPF);
 ecpf_err:
@@ -1180,8 +1234,11 @@ void mlx5_eswitch_disable_pf_vf_vports(struct mlx5_eswitch *esw)
 {
 	mlx5_eswitch_unload_vf_vports(esw, esw->esw_funcs.num_vfs);
 
-	if (mlx5_ecpf_vport_exists(esw->dev))
+	if (mlx5_ecpf_vport_exists(esw->dev)) {
+		if (mlx5_core_ec_sriov_enabled(esw->dev))
+			mlx5_eswitch_unload_ec_vf_vports(esw, esw->esw_funcs.num_vfs);
 		mlx5_eswitch_unload_vport(esw, MLX5_VPORT_ECPF);
+	}
 
 	host_pf_disable_hca(esw->dev);
 	mlx5_eswitch_unload_vport(esw, MLX5_VPORT_PF);
@@ -1225,6 +1282,9 @@ mlx5_eswitch_update_num_of_vfs(struct mlx5_eswitch *esw, int num_vfs)
 
 	esw->esw_funcs.num_vfs = MLX5_GET(query_esw_functions_out, out,
 					  host_params_context.host_num_of_vfs);
+	if (mlx5_core_ec_sriov_enabled(esw->dev))
+		esw->esw_funcs.num_ec_vfs = num_vfs;
+
 	kvfree(out);
 }
 
@@ -1332,9 +1392,9 @@ int mlx5_eswitch_enable_locked(struct mlx5_eswitch *esw, int num_vfs)
 
 	mlx5_eswitch_event_handlers_register(esw);
 
-	esw_info(esw->dev, "Enable: mode(%s), nvfs(%d), active vports(%d)\n",
+	esw_info(esw->dev, "Enable: mode(%s), nvfs(%d), necvfs(%d), active vports(%d)\n",
 		 esw->mode == MLX5_ESWITCH_LEGACY ? "LEGACY" : "OFFLOADS",
-		 esw->esw_funcs.num_vfs, esw->enabled_vports);
+		 esw->esw_funcs.num_vfs, esw->esw_funcs.num_ec_vfs, esw->enabled_vports);
 
 	mlx5_esw_mode_change_notify(esw, esw->mode);
 
@@ -1356,7 +1416,7 @@ int mlx5_eswitch_enable_locked(struct mlx5_eswitch *esw, int num_vfs)
 int mlx5_eswitch_enable(struct mlx5_eswitch *esw, int num_vfs)
 {
 	bool toggle_lag;
-	int ret;
+	int ret = 0;
 
 	if (!mlx5_esw_allowed(esw))
 		return 0;
@@ -1376,10 +1436,21 @@ int mlx5_eswitch_enable(struct mlx5_eswitch *esw, int num_vfs)
 
 		vport_events = (esw->mode == MLX5_ESWITCH_LEGACY) ?
 					MLX5_LEGACY_SRIOV_VPORT_EVENTS : MLX5_VPORT_UC_ADDR_CHANGE;
-		ret = mlx5_eswitch_load_vf_vports(esw, num_vfs, vport_events);
-		if (!ret)
-			esw->esw_funcs.num_vfs = num_vfs;
+		/* If this is the ECPF the number of host VFs is managed via the
+		 * eswitch function change event handler, and any num_vfs provided
+		 * here are intended to be EC VFs.
+		 */
+		if (!mlx5_core_is_ecpf(esw->dev)) {
+			ret = mlx5_eswitch_load_vf_vports(esw, num_vfs, vport_events);
+			if (!ret)
+				esw->esw_funcs.num_vfs = num_vfs;
+		} else if (mlx5_core_ec_sriov_enabled(esw->dev)) {
+			ret = mlx5_eswitch_load_ec_vf_vports(esw, num_vfs, vport_events);
+			if (!ret)
+				esw->esw_funcs.num_ec_vfs = num_vfs;
+		}
 	}
+
 	up_write(&esw->mode_lock);
 
 	if (toggle_lag)
@@ -1399,16 +1470,22 @@ void mlx5_eswitch_disable_sriov(struct mlx5_eswitch *esw, bool clear_vf)
 	/* If driver is unloaded, this function is called twice by remove_one()
 	 * and mlx5_unload(). Prevent the second call.
 	 */
-	if (!esw->esw_funcs.num_vfs && !clear_vf)
+	if (!esw->esw_funcs.num_vfs && !esw->esw_funcs.num_ec_vfs && !clear_vf)
 		goto unlock;
 
-	esw_info(esw->dev, "Unload vfs: mode(%s), nvfs(%d), active vports(%d)\n",
+	esw_info(esw->dev, "Unload vfs: mode(%s), nvfs(%d), necvfs(%d), active vports(%d)\n",
 		 esw->mode == MLX5_ESWITCH_LEGACY ? "LEGACY" : "OFFLOADS",
-		 esw->esw_funcs.num_vfs, esw->enabled_vports);
-
-	mlx5_eswitch_unload_vf_vports(esw, esw->esw_funcs.num_vfs);
-	if (clear_vf)
-		mlx5_eswitch_clear_vf_vports_info(esw);
+		 esw->esw_funcs.num_vfs, esw->esw_funcs.num_ec_vfs, esw->enabled_vports);
+
+	if (!mlx5_core_is_ecpf(esw->dev)) {
+		mlx5_eswitch_unload_vf_vports(esw, esw->esw_funcs.num_vfs);
+		if (clear_vf)
+			mlx5_eswitch_clear_vf_vports_info(esw);
+	} else if (mlx5_core_ec_sriov_enabled(esw->dev)) {
+		mlx5_eswitch_unload_ec_vf_vports(esw, esw->esw_funcs.num_ec_vfs);
+		if (clear_vf)
+			mlx5_eswitch_clear_ec_vf_vports_info(esw);
+	}
 
 	if (esw->mode == MLX5_ESWITCH_OFFLOADS) {
 		struct devlink *devlink = priv_to_devlink(esw->dev);
@@ -1419,7 +1496,10 @@ void mlx5_eswitch_disable_sriov(struct mlx5_eswitch *esw, bool clear_vf)
 	if (esw->mode == MLX5_ESWITCH_LEGACY)
 		mlx5_eswitch_disable_locked(esw);
 
-	esw->esw_funcs.num_vfs = 0;
+	if (!mlx5_core_is_ecpf(esw->dev))
+		esw->esw_funcs.num_vfs = 0;
+	else
+		esw->esw_funcs.num_ec_vfs = 0;
 
 unlock:
 	up_write(&esw->mode_lock);
@@ -1439,9 +1519,9 @@ void mlx5_eswitch_disable_locked(struct mlx5_eswitch *esw)
 
 	mlx5_eswitch_event_handlers_unregister(esw);
 
-	esw_info(esw->dev, "Disable: mode(%s), nvfs(%d), active vports(%d)\n",
+	esw_info(esw->dev, "Disable: mode(%s), nvfs(%d), necvfs(%d), active vports(%d)\n",
 		 esw->mode == MLX5_ESWITCH_LEGACY ? "LEGACY" : "OFFLOADS",
-		 esw->esw_funcs.num_vfs, esw->enabled_vports);
+		 esw->esw_funcs.num_vfs, esw->esw_funcs.num_ec_vfs, esw->enabled_vports);
 
 	if (esw->fdb_table.flags & MLX5_ESW_FDB_CREATED) {
 		esw->fdb_table.flags &= ~MLX5_ESW_FDB_CREATED;
@@ -1601,6 +1681,17 @@ static int mlx5_esw_vports_init(struct mlx5_eswitch *esw)
 		idx++;
 	}
 
+	if (mlx5_core_ec_sriov_enabled(esw->dev)) {
+		int ec_vf_base_num = mlx5_core_ec_vf_vport_base(dev);
+
+		for (i = 0; i < mlx5_core_max_ec_vfs(esw->dev); i++) {
+			err = mlx5_esw_vport_alloc(esw, idx, ec_vf_base_num + i);
+			if (err)
+				goto err;
+			idx++;
+		}
+	}
+
 	if (mlx5_ecpf_vport_exists(dev) ||
 	    mlx5_core_is_ecpf_esw_manager(dev)) {
 		err = mlx5_esw_vport_alloc(esw, idx, MLX5_VPORT_ECPF);
drivers/net/ethernet/mellanox/mlx5/core/eswitch.h +13 -0
index d3608f198e0a..266b60fefe25 100644
@@ -289,6 +289,7 @@ struct mlx5_host_work {
 struct mlx5_esw_functions {
 	struct mlx5_nb		nb;
 	u16			num_vfs;
+	u16			num_ec_vfs;
 };
 
 enum {
@@ -654,6 +655,18 @@ void mlx5e_tc_clean_fdb_peer_flows(struct mlx5_eswitch *esw);
 #define mlx5_esw_for_each_host_func_vport(esw, index, vport, last)	\
 	mlx5_esw_for_each_vport_marked(esw, index, vport, last, MLX5_ESW_VPT_HOST_FN)
 
+/* This macro should only be used if EC SRIOV is enabled.
+ *
+ * Because there were no more marks available on the xarray this uses a
+ * for_each_range approach. The range is only valid when EC SRIOV is enabled
+ */
+#define mlx5_esw_for_each_ec_vf_vport(esw, index, vport, last)		\
+	xa_for_each_range(&((esw)->vports),				\
+			  index,					\
+			  vport,					\
+			  MLX5_CAP_GEN_2((esw->dev), ec_vf_vport_base),	\
+			  (last) - 1)
+
 struct mlx5_eswitch *mlx5_devlink_eswitch_get(struct devlink *devlink);
 struct mlx5_vport *__must_check
 mlx5_eswitch_get_vport(struct mlx5_eswitch *esw, u16 vport_num);
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +22 -0
index 625982454575..68798aed792f 100644
@@ -3287,6 +3287,9 @@ int esw_offloads_enable(struct mlx5_eswitch *esw)
 	/* Representor will control the vport link state */
 	mlx5_esw_for_each_vf_vport(esw, i, vport, esw->esw_funcs.num_vfs)
 		vport->info.link_state = MLX5_VPORT_ADMIN_STATE_DOWN;
+	if (mlx5_core_ec_sriov_enabled(esw->dev))
+		mlx5_esw_for_each_ec_vf_vport(esw, i, vport, esw->esw_funcs.num_ec_vfs)
+			vport->info.link_state = MLX5_VPORT_ADMIN_STATE_DOWN;
 
 	/* Uplink vport rep must load first. */
 	err = esw_offloads_load_rep(esw, MLX5_VPORT_UPLINK);
@@ -3524,8 +3527,27 @@ static int mlx5_esw_vports_inline_set(struct mlx5_eswitch *esw, u8 mlx5_mode,
 			goto revert_inline_mode;
 		}
 	}
+	if (mlx5_core_ec_sriov_enabled(esw->dev)) {
+		mlx5_esw_for_each_ec_vf_vport(esw, i, vport, esw->esw_funcs.num_ec_vfs) {
+			err = mlx5_modify_nic_vport_min_inline(dev, vport->vport, mlx5_mode);
+			if (err) {
+				err_vport_num = vport->vport;
+				NL_SET_ERR_MSG_MOD(extack,
+						   "Failed to set min inline on vport");
+				goto revert_ec_vf_inline_mode;
+			}
+		}
+	}
 	return 0;
 
+revert_ec_vf_inline_mode:
+	mlx5_esw_for_each_ec_vf_vport(esw, i, vport, esw->esw_funcs.num_ec_vfs) {
+		if (vport->vport == err_vport_num)
+			break;
+		mlx5_modify_nic_vport_min_inline(dev,
+						 vport->vport,
+						 esw->offloads.inline_mode);
+	}
 revert_inline_mode:
 	mlx5_esw_for_each_host_func_vport(esw, i, vport, esw->esw_funcs.num_vfs) {
 		if (vport->vport == err_vport_num)
Signature · 1 line
2.40.1
#7 lore raw

PATCH 6/15 net/mlx5: Add/remove peer miss rules for EC VFs

In reply to Saeed Mahameed

From: Daniel Jurgens <[email protected]>

Add and remove the peer miss rules for EC VFs. It's possible that there are different amounts of total VFs per function so only create rules for the minimum number of max VFs.

---
 .../mellanox/mlx5/core/eswitch_offloads.c     | 32 +++++++++++++++++++
 1 file changed, 32 insertions(+)
drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c +32 -0
index 68798aed792f..fdf482f6fb34 100644
@@ -1125,11 +1125,32 @@ static int esw_add_fdb_peer_miss_rules(struct mlx5_eswitch *esw,
 		flows[vport->index] = flow;
 	}
 
+	if (mlx5_core_ec_sriov_enabled(esw->dev)) {
+		mlx5_esw_for_each_ec_vf_vport(esw, i, vport, mlx5_core_max_ec_vfs(esw->dev)) {
+			if (i >= mlx5_core_max_ec_vfs(peer_dev))
+				break;
+			esw_set_peer_miss_rule_source_port(esw, peer_dev->priv.eswitch,
+							   spec, vport->vport);
+			flow = mlx5_add_flow_rules(esw->fdb_table.offloads.slow_fdb,
+						   spec, &flow_act, &dest, 1);
+			if (IS_ERR(flow)) {
+				err = PTR_ERR(flow);
+				goto add_ec_vf_flow_err;
+			}
+			flows[vport->index] = flow;
+		}
+	}
 	esw->fdb_table.offloads.peer_miss_rules[mlx5_get_dev_index(peer_dev)] = flows;
 
 	kvfree(spec);
 	return 0;
 
+add_ec_vf_flow_err:
+	mlx5_esw_for_each_ec_vf_vport(esw, i, vport, mlx5_core_max_ec_vfs(esw->dev)) {
+		if (!flows[vport->index])
+			continue;
+		mlx5_del_flow_rules(flows[vport->index]);
+	}
 add_vf_flow_err:
 	mlx5_esw_for_each_vf_vport(esw, i, vport, mlx5_core_max_vfs(esw->dev)) {
 		if (!flows[vport->index])
@@ -1162,6 +1183,17 @@ static void esw_del_fdb_peer_miss_rules(struct mlx5_eswitch *esw,
 
 	flows = esw->fdb_table.offloads.peer_miss_rules[mlx5_get_dev_index(peer_dev)];
 
+	if (mlx5_core_ec_sriov_enabled(esw->dev)) {
+		mlx5_esw_for_each_ec_vf_vport(esw, i, vport, mlx5_core_max_ec_vfs(esw->dev)) {
+			/* The flow for a particular vport could be NULL if the other ECPF
+			 * has fewer or no VFs enabled
+			 */
+			if (!flows[vport->index])
+				continue;
+			mlx5_del_flow_rules(flows[vport->index]);
+		}
+	}
+
 	mlx5_esw_for_each_vf_vport(esw, i, vport, mlx5_core_max_vfs(esw->dev))
 		mlx5_del_flow_rules(flows[vport->index]);
 
Signature · 1 line
2.40.1
#8 lore raw

PATCH 7/15 net/mlx5: Add new page type for EC VF pages

In reply to Saeed Mahameed

From: Daniel Jurgens <[email protected]>

When the embedded cpu supports SRIOV it can be enabled and disabled independently from the host SRIOV. Track the pages separately so we can properly wait for returned VF pages.

---
 drivers/net/ethernet/mellanox/mlx5/core/debugfs.c   |  1 +
 drivers/net/ethernet/mellanox/mlx5/core/pagealloc.c | 11 ++++++++++-
 include/linux/mlx5/driver.h                         |  1 +
 3 files changed, 12 insertions(+), 1 deletion(-)
drivers/net/ethernet/mellanox/mlx5/core/debugfs.c +1 -0
index bb95b40d25eb..fc13b41cc9b2 100644
@@ -246,6 +246,7 @@ void mlx5_pages_debugfs_init(struct mlx5_core_dev *dev)
 
 	debugfs_create_u32("fw_pages_total", 0400, pages, &dev->priv.fw_pages);
 	debugfs_create_u32("fw_pages_vfs", 0400, pages, &dev->priv.page_counters[MLX5_VF]);
+	debugfs_create_u32("fw_pages_ec_vfs", 0400, pages, &dev->priv.page_counters[MLX5_EC_VF]);
 	debugfs_create_u32("fw_pages_sfs", 0400, pages, &dev->priv.page_counters[MLX5_SF]);
 	debugfs_create_u32("fw_pages_host_pf", 0400, pages, &dev->priv.page_counters[MLX5_HOST_PF]);
 	debugfs_create_u32("fw_pages_alloc_failed", 0400, pages, &dev->priv.fw_pages_alloc_failed);
drivers/net/ethernet/mellanox/mlx5/core/pagealloc.c +10 -1
index 95dc67fb3001..dcf58efac159 100644
@@ -79,7 +79,13 @@ static u16 func_id_to_type(struct mlx5_core_dev *dev, u16 func_id, bool ec_funct
 	if (!func_id)
 		return mlx5_core_is_ecpf(dev) && !ec_function ? MLX5_HOST_PF : MLX5_PF;
 
-	return func_id <= mlx5_core_max_vfs(dev) ?  MLX5_VF : MLX5_SF;
+	if (func_id <= max(mlx5_core_max_vfs(dev), mlx5_core_max_ec_vfs(dev))) {
+		if (ec_function)
+			return MLX5_EC_VF;
+		else
+			return MLX5_VF;
+	}
+	return MLX5_SF;
 }
 
 static u32 mlx5_get_ec_function(u32 function)
@@ -730,6 +736,9 @@ int mlx5_reclaim_startup_pages(struct mlx5_core_dev *dev)
 	WARN(dev->priv.page_counters[MLX5_HOST_PF],
 	     "External host PF FW pages counter is %d after reclaiming all pages\n",
 	     dev->priv.page_counters[MLX5_HOST_PF]);
+	WARN(dev->priv.page_counters[MLX5_EC_VF],
+	     "EC VFs FW pages counter is %d after reclaiming all pages\n",
+	     dev->priv.page_counters[MLX5_EC_VF]);
 
 	return 0;
 }
include/linux/mlx5/driver.h +1 -0
index 252b6a6965b8..18a608a1f567 100644
@@ -581,6 +581,7 @@ enum mlx5_func_type {
 	MLX5_VF,
 	MLX5_SF,
 	MLX5_HOST_PF,
+	MLX5_EC_VF,
 	MLX5_FUNC_TYPE_NUM,
 };
 
Signature · 1 line
2.40.1
#9 lore raw

PATCH 8/15 net/mlx5: Use correct vport when restoring GUIDs

In reply to Saeed Mahameed

From: Daniel Jurgens <[email protected]>

Prior to enabling EC VF functionality the vport number and function ID were always the same. That's not the case now. Use the correct vport number to modify the HCA vport context.

---
 drivers/net/ethernet/mellanox/mlx5/core/sriov.c | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/sriov.c +7 -3
index f07d00929162..c2463a1d7035 100644
@@ -37,7 +37,7 @@
 #include "mlx5_irq.h"
 #include "eswitch.h"
 
-static int sriov_restore_guids(struct mlx5_core_dev *dev, int vf)
+static int sriov_restore_guids(struct mlx5_core_dev *dev, int vf, u16 func_id)
 {
 	struct mlx5_core_sriov *sriov = &dev->priv.sriov;
 	struct mlx5_hca_vport_context *in;
@@ -59,7 +59,7 @@ static int sriov_restore_guids(struct mlx5_core_dev *dev, int vf)
 			!!(in->node_guid) * MLX5_HCA_VPORT_SEL_NODE_GUID |
 			!!(in->policy) * MLX5_HCA_VPORT_SEL_STATE_POLICY;
 
-		err = mlx5_core_modify_hca_vport_context(dev, 1, 1, vf + 1, in);
+		err = mlx5_core_modify_hca_vport_context(dev, 1, 1, func_id, in);
 		if (err)
 			mlx5_core_warn(dev, "modify vport context failed, unable to restore VF %d settings\n", vf);
 
@@ -73,6 +73,7 @@ static int mlx5_device_enable_sriov(struct mlx5_core_dev *dev, int num_vfs)
 {
 	struct mlx5_core_sriov *sriov = &dev->priv.sriov;
 	int err, vf, num_msix_count;
+	int vport_num;
 
 	err = mlx5_eswitch_enable(dev->priv.eswitch, num_vfs);
 	if (err) {
@@ -104,7 +105,10 @@ static int mlx5_device_enable_sriov(struct mlx5_core_dev *dev, int num_vfs)
 
 		sriov->vfs_ctx[vf].enabled = 1;
 		if (MLX5_CAP_GEN(dev, port_type) == MLX5_CAP_PORT_TYPE_IB) {
-			err = sriov_restore_guids(dev, vf);
+			vport_num = mlx5_core_ec_sriov_enabled(dev) ?
+					mlx5_core_ec_vf_vport_base(dev) + vf
+					: vf + 1;
+			err = sriov_restore_guids(dev, vf, vport_num);
 			if (err) {
 				mlx5_core_warn(dev,
 					       "failed to restore VF %d settings, err %d\n",
Signature · 1 line
2.40.1
#10 lore raw

PATCH 9/15 net/mlx5: Query correct caps for min msix vectors

In reply to Saeed Mahameed

From: Daniel Jurgens <[email protected]>

The VFs on the host and the embedded CPU platform share function numbers. Set the ec_vf_function field to query the caps for the correct function.

---
 .../net/ethernet/mellanox/mlx5/core/pci_irq.c    | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)
drivers/net/ethernet/mellanox/mlx5/core/pci_irq.c +15 -1
index 843da89a9035..b2dbae763ca6 100644
@@ -41,6 +41,15 @@ struct mlx5_irq_table {
 	struct mlx5_irq_pool *sf_comp_pool;
 };
 
+static int mlx5_core_func_to_vport(const struct mlx5_core_dev *dev,
+				   int func,
+				   bool ec_vf_func)
+{
+	if (!ec_vf_func)
+		return func;
+	return mlx5_core_ec_vf_vport_base(dev) + func - 1;
+}
+
 /**
  * mlx5_get_default_msix_vec_count - Get the default number of MSI-X vectors
  *                                   to be ssigned to each VF.
@@ -79,6 +88,8 @@ int mlx5_set_msix_vec_count(struct mlx5_core_dev *dev, int function_id,
 	int set_sz = MLX5_ST_SZ_BYTES(set_hca_cap_in);
 	void *hca_cap = NULL, *query_cap = NULL, *cap;
 	int num_vf_msix, min_msix, max_msix;
+	bool ec_vf_function;
+	int vport;
 	int ret;
 
 	num_vf_msix = MLX5_CAP_GEN_MAX(dev, num_total_dynamic_vf_msix);
@@ -104,7 +115,9 @@ int mlx5_set_msix_vec_count(struct mlx5_core_dev *dev, int function_id,
 		goto out;
 	}
 
-	ret = mlx5_vport_get_other_func_general_cap(dev, function_id, query_cap);
+	ec_vf_function = mlx5_core_ec_sriov_enabled(dev);
+	vport = mlx5_core_func_to_vport(dev, function_id, ec_vf_function);
+	ret = mlx5_vport_get_other_func_general_cap(dev, vport, query_cap);
 	if (ret)
 		goto out;
 
@@ -115,6 +128,7 @@ int mlx5_set_msix_vec_count(struct mlx5_core_dev *dev, int function_id,
 
 	MLX5_SET(set_hca_cap_in, hca_cap, opcode, MLX5_CMD_OP_SET_HCA_CAP);
 	MLX5_SET(set_hca_cap_in, hca_cap, other_function, 1);
+	MLX5_SET(set_hca_cap_in, hca_cap, ec_vf_function, ec_vf_function);
 	MLX5_SET(set_hca_cap_in, hca_cap, function_id, function_id);
 
 	MLX5_SET(set_hca_cap_in, hca_cap, op_mod,
Signature · 1 line
2.40.1
#11 lore raw

PATCH 10/15 net/mlx5: Update SRIOV enable/disable to handle EC/VFs

In reply to Saeed Mahameed

From: Daniel Jurgens <[email protected]>

Previously on the embedded CPU platform SRIOV was never enabled/disabled via mlx5_core_sriov_configure. Host VF updates are provided by an event handler. Now in the disable flow it must be known if this is a disable due to driver unload or SRIOV detach, or if the user updated the number of VFs. If due to change in the number of VFs only wait for the pages of ECVFs.

---
 .../net/ethernet/mellanox/mlx5/core/main.c    |  2 +-
 .../ethernet/mellanox/mlx5/core/mlx5_core.h   |  2 +-
 .../net/ethernet/mellanox/mlx5/core/sriov.c   | 35 +++++++++++++++----
 3 files changed, 30 insertions(+), 9 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/main.c +1 -1
index d6ee016deae1..fed8b48a5b20 100644
@@ -1809,7 +1809,7 @@ static void remove_one(struct pci_dev *pdev)
 	mlx5_drain_fw_reset(dev);
 	mlx5_drain_health_wq(dev);
 	devlink_unregister(devlink);
-	mlx5_sriov_disable(pdev);
+	mlx5_sriov_disable(pdev, false);
 	mlx5_thermal_uninit(dev);
 	mlx5_crdump_disable(dev);
 	mlx5_uninit_one(dev);
drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h +1 -1
index 7ca0c7a547aa..7a5f04082058 100644
@@ -195,7 +195,7 @@ void mlx5_sriov_cleanup(struct mlx5_core_dev *dev);
 int mlx5_sriov_attach(struct mlx5_core_dev *dev);
 void mlx5_sriov_detach(struct mlx5_core_dev *dev);
 int mlx5_core_sriov_configure(struct pci_dev *dev, int num_vfs);
-void mlx5_sriov_disable(struct pci_dev *pdev);
+void mlx5_sriov_disable(struct pci_dev *pdev, bool num_vf_change);
 int mlx5_core_sriov_set_msix_vec_count(struct pci_dev *vf, int msix_vec_count);
 int mlx5_core_enable_hca(struct mlx5_core_dev *dev, u16 func_id);
 int mlx5_core_disable_hca(struct mlx5_core_dev *dev, u16 func_id);
drivers/net/ethernet/mellanox/mlx5/core/sriov.c +28 -7
index c2463a1d7035..b73583b0a0fe 100644
@@ -123,9 +123,11 @@ static int mlx5_device_enable_sriov(struct mlx5_core_dev *dev, int num_vfs)
 }
 
 static void
-mlx5_device_disable_sriov(struct mlx5_core_dev *dev, int num_vfs, bool clear_vf)
+mlx5_device_disable_sriov(struct mlx5_core_dev *dev, int num_vfs, bool clear_vf, bool num_vf_change)
 {
 	struct mlx5_core_sriov *sriov = &dev->priv.sriov;
+	bool wait_for_ec_vf_pages = true;
+	bool wait_for_vf_pages = true;
 	int err;
 	int vf;
 
@@ -147,11 +149,30 @@ mlx5_device_disable_sriov(struct mlx5_core_dev *dev, int num_vfs, bool clear_vf)
 
 	mlx5_eswitch_disable_sriov(dev->priv.eswitch, clear_vf);
 
+	/* There are a number of scenarios when SRIOV is being disabled:
+	 *     1. VFs or ECVFs had been created, and now set back to 0 (num_vf_change == true).
+	 *		- If EC SRIOV is enabled then this flow is happening on the
+	 *		  embedded platform, wait for only EC VF pages.
+	 *		- If EC SRIOV is not enabled this flow is happening on non-embedded
+	 *		  platform, wait for the VF pages.
+	 *
+	 *     2. The driver is being unloaded. In this case wait for all pages.
+	 */
+	if (num_vf_change) {
+		if (mlx5_core_ec_sriov_enabled(dev))
+			wait_for_vf_pages = false;
+		else
+			wait_for_ec_vf_pages = false;
+	}
+
+	if (wait_for_ec_vf_pages && mlx5_wait_for_pages(dev, &dev->priv.page_counters[MLX5_EC_VF]))
+		mlx5_core_warn(dev, "timeout reclaiming EC VFs pages\n");
+
 	/* For ECPFs, skip waiting for host VF pages until ECPF is destroyed */
 	if (mlx5_core_is_ecpf(dev))
 		return;
 
-	if (mlx5_wait_for_pages(dev, &dev->priv.page_counters[MLX5_VF]))
+	if (wait_for_vf_pages && mlx5_wait_for_pages(dev, &dev->priv.page_counters[MLX5_VF]))
 		mlx5_core_warn(dev, "timeout reclaiming VFs pages\n");
 }
 
@@ -172,12 +193,12 @@ static int mlx5_sriov_enable(struct pci_dev *pdev, int num_vfs)
 	err = pci_enable_sriov(pdev, num_vfs);
 	if (err) {
 		mlx5_core_warn(dev, "pci_enable_sriov failed : %d\n", err);
-		mlx5_device_disable_sriov(dev, num_vfs, true);
+		mlx5_device_disable_sriov(dev, num_vfs, true, true);
 	}
 	return err;
 }
 
-void mlx5_sriov_disable(struct pci_dev *pdev)
+void mlx5_sriov_disable(struct pci_dev *pdev, bool num_vf_change)
 {
 	struct mlx5_core_dev *dev  = pci_get_drvdata(pdev);
 	struct devlink *devlink = priv_to_devlink(dev);
@@ -185,7 +206,7 @@ void mlx5_sriov_disable(struct pci_dev *pdev)
 
 	pci_disable_sriov(pdev);
 	devl_lock(devlink);
-	mlx5_device_disable_sriov(dev, num_vfs, true);
+	mlx5_device_disable_sriov(dev, num_vfs, true, num_vf_change);
 	devl_unlock(devlink);
 }
 
@@ -200,7 +221,7 @@ int mlx5_core_sriov_configure(struct pci_dev *pdev, int num_vfs)
 	if (num_vfs)
 		err = mlx5_sriov_enable(pdev, num_vfs);
 	else
-		mlx5_sriov_disable(pdev);
+		mlx5_sriov_disable(pdev, true);
 
 	if (!err)
 		sriov->num_vfs = num_vfs;
@@ -245,7 +266,7 @@ void mlx5_sriov_detach(struct mlx5_core_dev *dev)
 	if (!mlx5_core_is_pf(dev))
 		return;
 
-	mlx5_device_disable_sriov(dev, pci_num_vf(dev->pdev), false);
+	mlx5_device_disable_sriov(dev, pci_num_vf(dev->pdev), false, false);
 }
 
 static u16 mlx5_get_max_vfs(struct mlx5_core_dev *dev)
Signature · 1 line
2.40.1
#12 lore raw

PATCH 11/15 net/mlx5: Set max number of embedded CPU VFs

In reply to Saeed Mahameed

From: Daniel Jurgens <[email protected]>

Set the maximum number of embedded cpu VF functions available.

---
 drivers/net/ethernet/mellanox/mlx5/core/sriov.c | 1 +
 1 file changed, 1 insertion(+)
drivers/net/ethernet/mellanox/mlx5/core/sriov.c +1 -0
index b73583b0a0fe..4e42a3b9b8ee 100644
@@ -305,6 +305,7 @@ int mlx5_sriov_init(struct mlx5_core_dev *dev)
 	total_vfs = pci_sriov_get_totalvfs(pdev);
 	sriov->max_vfs = mlx5_get_max_vfs(dev);
 	sriov->num_vfs = pci_num_vf(pdev);
+	sriov->max_ec_vfs = mlx5_core_ec_sriov_enabled(dev) ? pci_sriov_get_totalvfs(dev->pdev) : 0;
 	sriov->vfs_ctx = kcalloc(total_vfs, sizeof(*sriov->vfs_ctx), GFP_KERNEL);
 	if (!sriov->vfs_ctx)
 		return -ENOMEM;
Signature · 1 line
2.40.1
#13 lore raw

PATCH 12/15 net/mlx5: Split function_setup() to enable and open functions

In reply to Saeed Mahameed

From: Shay Drory <[email protected]>

mlx5_cmd_init_hca() is taking ~0.2 seconds. In case of a user who desire to disable some of the SF aux devices, and with large scale-1K SFs for example, this user will waste more than 3 minutes on mlx5_cmd_init_hca() which isn't needed at this stage.

Downstream patch will change SFs which are probe over the E-switch, local SFs, to be probed without any aux dev. In order to support this, split function_setup() to avoid executing mlx5_cmd_init_hca().

---
 .../net/ethernet/mellanox/mlx5/core/main.c    | 83 +++++++++++++------
 1 file changed, 58 insertions(+), 25 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/main.c +58 -25
index fed8b48a5b20..0faae77d84e6 100644
@@ -1118,7 +1118,7 @@ static void mlx5_cleanup_once(struct mlx5_core_dev *dev)
 	mlx5_devcom_unregister_device(dev->priv.devcom);
 }
 
-static int mlx5_function_setup(struct mlx5_core_dev *dev, bool boot, u64 timeout)
+static int mlx5_function_enable(struct mlx5_core_dev *dev, bool boot, u64 timeout)
 {
 	int err;
 
@@ -1183,28 +1183,56 @@ static int mlx5_function_setup(struct mlx5_core_dev *dev, bool boot, u64 timeout
 		goto reclaim_boot_pages;
 	}
 
+	return 0;
+
+reclaim_boot_pages:
+	mlx5_reclaim_startup_pages(dev);
+err_disable_hca:
+	mlx5_core_disable_hca(dev, 0);
+stop_health_poll:
+	mlx5_stop_health_poll(dev, boot);
+err_cmd_cleanup:
+	mlx5_cmd_set_state(dev, MLX5_CMDIF_STATE_DOWN);
+	mlx5_cmd_cleanup(dev);
+
+	return err;
+}
+
+static void mlx5_function_disable(struct mlx5_core_dev *dev, bool boot)
+{
+	mlx5_reclaim_startup_pages(dev);
+	mlx5_core_disable_hca(dev, 0);
+	mlx5_stop_health_poll(dev, boot);
+	mlx5_cmd_set_state(dev, MLX5_CMDIF_STATE_DOWN);
+	mlx5_cmd_cleanup(dev);
+}
+
+static int mlx5_function_open(struct mlx5_core_dev *dev)
+{
+	int err;
+
 	err = set_hca_ctrl(dev);
 	if (err) {
 		mlx5_core_err(dev, "set_hca_ctrl failed\n");
-		goto reclaim_boot_pages;
+		return err;
 	}
 
 	err = set_hca_cap(dev);
 	if (err) {
 		mlx5_core_err(dev, "set_hca_cap failed\n");
-		goto reclaim_boot_pages;
+		return err;
 	}
 
 	err = mlx5_satisfy_startup_pages(dev, 0);
 	if (err) {
 		mlx5_core_err(dev, "failed to allocate init pages\n");
-		goto reclaim_boot_pages;
+		return err;
 	}
 
 	err = mlx5_cmd_init_hca(dev, sw_owner_id);
 	if (err) {
 		mlx5_core_err(dev, "init hca failed\n");
-		goto reclaim_boot_pages;
+		return err;
 	}
 
 	mlx5_set_driver_version(dev);
@@ -1212,26 +1240,13 @@ static int mlx5_function_setup(struct mlx5_core_dev *dev, bool boot, u64 timeout
 	err = mlx5_query_hca_caps(dev);
 	if (err) {
 		mlx5_core_err(dev, "query hca failed\n");
-		goto reclaim_boot_pages;
+		return err;
 	}
 	mlx5_start_health_fw_log_up(dev);
-
 	return 0;
-
-reclaim_boot_pages:
-	mlx5_reclaim_startup_pages(dev);
-err_disable_hca:
-	mlx5_core_disable_hca(dev, 0);
-stop_health_poll:
-	mlx5_stop_health_poll(dev, boot);
-err_cmd_cleanup:
-	mlx5_cmd_set_state(dev, MLX5_CMDIF_STATE_DOWN);
-	mlx5_cmd_cleanup(dev);
-
-	return err;
 }
 
-static int mlx5_function_teardown(struct mlx5_core_dev *dev, bool boot)
+static int mlx5_function_close(struct mlx5_core_dev *dev)
 {
 	int err;
 
@@ -1240,15 +1255,33 @@ static int mlx5_function_teardown(struct mlx5_core_dev *dev, bool boot)
 		mlx5_core_err(dev, "tear_down_hca failed, skip cleanup\n");
 		return err;
 	}
-	mlx5_reclaim_startup_pages(dev);
-	mlx5_core_disable_hca(dev, 0);
-	mlx5_stop_health_poll(dev, boot);
-	mlx5_cmd_set_state(dev, MLX5_CMDIF_STATE_DOWN);
-	mlx5_cmd_cleanup(dev);
 
 	return 0;
 }
 
+static int mlx5_function_setup(struct mlx5_core_dev *dev, bool boot, u64 timeout)
+{
+	int err;
+
+	err = mlx5_function_enable(dev, boot, timeout);
+	if (err)
+		return err;
+
+	err = mlx5_function_open(dev);
+	if (err)
+		mlx5_function_disable(dev, boot);
+	return err;
+}
+
+static int mlx5_function_teardown(struct mlx5_core_dev *dev, bool boot)
+{
+	int err = mlx5_function_close(dev);
+
+	if (!err)
+		mlx5_function_disable(dev, boot);
+	return err;
+}
+
 static int mlx5_load(struct mlx5_core_dev *dev)
 {
 	int err;
Signature · 1 line
2.40.1
#14 lore raw

PATCH 13/15 net/mlx5: Move esw multiport devlink param to eswitch code

In reply to Saeed Mahameed

From: Shay Drory <[email protected]>

Move the param registration and handling code into the eswitch code as they are related to each other. No point in having the devlink param registration done in separate file.

---
 .../net/ethernet/mellanox/mlx5/core/devlink.c | 34 -------------
 .../net/ethernet/mellanox/mlx5/core/eswitch.c | 49 ++++++++++++++++++-
 2 files changed, 47 insertions(+), 36 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/devlink.c +0 -34
index 63635cc44479..27197acdb4d8 100644
@@ -7,7 +7,6 @@
 #include "fw_reset.h"
 #include "fs_core.h"
 #include "eswitch.h"
-#include "lag/lag.h"
 #include "esw/qos.h"
 #include "sf/dev/dev.h"
 #include "sf/sf.h"
@@ -427,33 +426,6 @@ static int mlx5_devlink_large_group_num_validate(struct devlink *devlink, u32 id
 
 	return 0;
 }
-
-static int mlx5_devlink_esw_multiport_set(struct devlink *devlink, u32 id,
-					  struct devlink_param_gset_ctx *ctx)
-{
-	struct mlx5_core_dev *dev = devlink_priv(devlink);
-
-	if (!MLX5_ESWITCH_MANAGER(dev))
-		return -EOPNOTSUPP;
-
-	if (ctx->val.vbool)
-		return mlx5_lag_mpesw_enable(dev);
-
-	mlx5_lag_mpesw_disable(dev);
-	return 0;
-}
-
-static int mlx5_devlink_esw_multiport_get(struct devlink *devlink, u32 id,
-					  struct devlink_param_gset_ctx *ctx)
-{
-	struct mlx5_core_dev *dev = devlink_priv(devlink);
-
-	if (!MLX5_ESWITCH_MANAGER(dev))
-		return -EOPNOTSUPP;
-
-	ctx->val.vbool = mlx5_lag_is_mpesw(dev);
-	return 0;
-}
 #endif
 
 static int mlx5_devlink_eq_depth_validate(struct devlink *devlink, u32 id,
@@ -527,12 +499,6 @@ static const struct devlink_param mlx5_devlink_params[] = {
 			     BIT(DEVLINK_PARAM_CMODE_DRIVERINIT),
 			     NULL, NULL,
 			     mlx5_devlink_large_group_num_validate),
-	DEVLINK_PARAM_DRIVER(MLX5_DEVLINK_PARAM_ID_ESW_MULTIPORT,
-			     "esw_multiport", DEVLINK_PARAM_TYPE_BOOL,
-			     BIT(DEVLINK_PARAM_CMODE_RUNTIME),
-			     mlx5_devlink_esw_multiport_get,
-			     mlx5_devlink_esw_multiport_set,
-			     NULL),
 #endif
 	DEVLINK_PARAM_GENERIC(IO_EQ_SIZE, BIT(DEVLINK_PARAM_CMODE_DRIVERINIT),
 			      NULL, NULL, mlx5_devlink_eq_depth_validate),
drivers/net/ethernet/mellanox/mlx5/core/eswitch.c +47 -2

80 lines not on this page · raw message

Signature · 1 line
2.40.1
#15 lore raw

PATCH 14/15 net/mlx5: Light probe local SFs

In reply to Saeed Mahameed

From: Shay Drory <[email protected]>

In case user wants to configure the SFs, for example: to use only vdpa functionality, he needs to fully probe a SF, configure what he wants, and afterward reload the SF.

In order to save the time of the reload, local SFs will probe without any auxiliary sub-device, so that the SFs can be configured prior to its full probe.

The defaults of the enable_* devlink params of these SFs are set to false.

Usage example:
Create SF:

$ devlink port add pci/0000:08:00.0 flavour pcisf pfnum 0 sfnum 11
$ devlink port function set pci/0000:08:00.0/32768 \
               hw_addr 00:00:00:00:00:11 state active

Enable ETH auxiliary device:

$ devlink dev param set auxiliary/mlx5_core.sf.1 \
              name enable_eth value true cmode driverinit

Now, in order to fully probe the SF, use devlink reload:

$ devlink dev reload auxiliary/mlx5_core.sf.1

At this point the user have SF devlink instance with auxiliary device for the Ethernet functionality only.

---
 .../ethernet/mellanox/mlx5/switchdev.rst      |  20 +++
 drivers/net/ethernet/mellanox/mlx5/core/dev.c |  16 +++
 .../net/ethernet/mellanox/mlx5/core/devlink.c |  20 ++-
 .../net/ethernet/mellanox/mlx5/core/health.c  |  24 ++--
 .../net/ethernet/mellanox/mlx5/core/main.c    | 124 ++++++++++++++++--
 .../ethernet/mellanox/mlx5/core/mlx5_core.h   |   7 +
 .../mellanox/mlx5/core/sf/dev/driver.c        |  15 ++-
 7 files changed, 203 insertions(+), 23 deletions(-)
Documentation/networking/device_drivers/ethernet/mellanox/mlx5/switchdev.rst +20 -0
index 01deedb71597..db62187eebce 100644
@@ -45,6 +45,26 @@ Following bridge VLAN functions are supported by mlx5:
 Subfunction
 ===========
 
+Subfunction which are spawned over the E-switch are created only with devlink
+device, and by default all the SF auxiliary devices are disabled.
+This will allow user to configure the SF before the SF have been fully probed,
+which will save time.
+
+Usage example:
+Create SF:
+$ devlink port add pci/0000:08:00.0 flavour pcisf pfnum 0 sfnum 11
+$ devlink port function set pci/0000:08:00.0/32768 \
+               hw_addr 00:00:00:00:00:11 state active
+
+Enable ETH auxiliary device:
+$ devlink dev param set auxiliary/mlx5_core.sf.1 \
+              name enable_eth value true cmode driverinit
+
+Now, in order to fully probe the SF, use devlink reload:
+$ devlink dev reload auxiliary/mlx5_core.sf.1
+
+mlx5 supports ETH,rdma and vdpa (vnet) auxiliary devices devlink params (see :ref:`Documentation/networking/devlink/devlink-params.rst`)
+
 mlx5 supports subfunction management using devlink port (see :ref:`Documentation/networking/devlink/devlink-port.rst <devlink_port>`) interface.
 
 A subfunction has its own function capabilities and its own resources. This
drivers/net/ethernet/mellanox/mlx5/core/dev.c +16 -0
index 1b33533b15de..617ac7e5d75c 100644
@@ -323,6 +323,18 @@ static void del_adev(struct auxiliary_device *adev)
 	auxiliary_device_uninit(adev);
 }
 
+void mlx5_dev_set_lightweight(struct mlx5_core_dev *dev)
+{
+	mutex_lock(&mlx5_intf_mutex);
+	dev->priv.flags |= MLX5_PRIV_FLAGS_DISABLE_ALL_ADEV;
+	mutex_unlock(&mlx5_intf_mutex);
+}
+
+bool mlx5_dev_is_lightweight(struct mlx5_core_dev *dev)
+{
+	return dev->priv.flags & MLX5_PRIV_FLAGS_DISABLE_ALL_ADEV;
+}
+
 int mlx5_attach_device(struct mlx5_core_dev *dev)
 {
 	struct mlx5_priv *priv = &dev->priv;
@@ -457,6 +469,10 @@ static int add_drivers(struct mlx5_core_dev *dev)
 		if (priv->adev[i])
 			continue;
 
+		if (mlx5_adev_devices[i].is_enabled &&
+		    !(mlx5_adev_devices[i].is_enabled(dev)))
+			continue;
+
 		if (mlx5_adev_devices[i].is_supported)
 			is_supported = mlx5_adev_devices[i].is_supported(dev);
 
drivers/net/ethernet/mellanox/mlx5/core/devlink.c +16 -4
index 27197acdb4d8..3d82ec890666 100644
@@ -141,6 +141,13 @@ static int mlx5_devlink_reload_down(struct devlink *devlink, bool netns_change,
 	bool sf_dev_allocated;
 	int ret = 0;
 
+	if (mlx5_dev_is_lightweight(dev)) {
+		if (action != DEVLINK_RELOAD_ACTION_DRIVER_REINIT)
+			return -EOPNOTSUPP;
+		mlx5_unload_one_light(dev);
+		return 0;
+	}
+
 	sf_dev_allocated = mlx5_sf_dev_allocated(dev);
 	if (sf_dev_allocated) {
 		/* Reload results in deleting SF device which further results in
@@ -193,6 +200,10 @@ static int mlx5_devlink_reload_up(struct devlink *devlink, enum devlink_reload_a
 	*actions_performed = BIT(action);
 	switch (action) {
 	case DEVLINK_RELOAD_ACTION_DRIVER_REINIT:
+		if (mlx5_dev_is_lightweight(dev)) {
+			mlx5_fw_reporters_create(dev);
+			return mlx5_init_one_devl_locked(dev);
+		}
 		ret = mlx5_load_one_devl_locked(dev, false);
 		break;
 	case DEVLINK_RELOAD_ACTION_FW_ACTIVATE:
@@ -511,7 +522,7 @@ static void mlx5_devlink_set_params_init_values(struct devlink *devlink)
 	struct mlx5_core_dev *dev = devlink_priv(devlink);
 	union devlink_param_value value;
 
-	value.vbool = MLX5_CAP_GEN(dev, roce);
+	value.vbool = MLX5_CAP_GEN(dev, roce) && !mlx5_dev_is_lightweight(dev);
 	devl_param_driverinit_value_set(devlink,
 					DEVLINK_PARAM_GENERIC_ID_ENABLE_ROCE,
 					value);
@@ -561,7 +572,7 @@ static int mlx5_devlink_eth_params_register(struct devlink *devlink)
 	if (err)
 		return err;
 
-	value.vbool = true;
+	value.vbool = !mlx5_dev_is_lightweight(dev);
 	devl_param_driverinit_value_set(devlink,
 					DEVLINK_PARAM_GENERIC_ID_ENABLE_ETH,
 					value);
@@ -601,6 +612,7 @@ static const struct devlink_param mlx5_devlink_rdma_params[] = {
 
 static int mlx5_devlink_rdma_params_register(struct devlink *devlink)
 {
+	struct mlx5_core_dev *dev = devlink_priv(devlink);
 	union devlink_param_value value;
 	int err;
 
@@ -612,7 +624,7 @@ static int mlx5_devlink_rdma_params_register(struct devlink *devlink)
 	if (err)
 		return err;
 
-	value.vbool = true;
+	value.vbool = !mlx5_dev_is_lightweight(dev);
 	devl_param_driverinit_value_set(devlink,
 					DEVLINK_PARAM_GENERIC_ID_ENABLE_RDMA,
 					value);
@@ -647,7 +659,7 @@ static int mlx5_devlink_vnet_params_register(struct devlink *devlink)
 	if (err)
 		return err;
 
-	value.vbool = true;
+	value.vbool = !mlx5_dev_is_lightweight(dev);
 	devl_param_driverinit_value_set(devlink,
 					DEVLINK_PARAM_GENERIC_ID_ENABLE_VNET,
 					value);
drivers/net/ethernet/mellanox/mlx5/core/health.c +15 -9
index 871c32dda66e..210100a4064a 100644
@@ -719,7 +719,7 @@ static const struct devlink_health_reporter_ops mlx5_fw_fatal_reporter_ops = {
 #define MLX5_FW_REPORTER_VF_GRACEFUL_PERIOD 30000
 #define MLX5_FW_REPORTER_DEFAULT_GRACEFUL_PERIOD MLX5_FW_REPORTER_VF_GRACEFUL_PERIOD
 
-static void mlx5_fw_reporters_create(struct mlx5_core_dev *dev)
+void mlx5_fw_reporters_create(struct mlx5_core_dev *dev)
 {
 	struct mlx5_core_health *health = &dev->priv.health;
 	struct devlink *devlink = priv_to_devlink(dev);
@@ -735,17 +735,17 @@ static void mlx5_fw_reporters_create(struct mlx5_core_dev *dev)
 	}
 
 	health->fw_reporter =
-		devlink_health_reporter_create(devlink, &mlx5_fw_reporter_ops,
-					       0, dev);
+		devl_health_reporter_create(devlink, &mlx5_fw_reporter_ops,
+					    0, dev);
 	if (IS_ERR(health->fw_reporter))
 		mlx5_core_warn(dev, "Failed to create fw reporter, err = %ld\n",
 			       PTR_ERR(health->fw_reporter));
 
 	health->fw_fatal_reporter =
-		devlink_health_reporter_create(devlink,
-					       &mlx5_fw_fatal_reporter_ops,
-					       grace_period,
-					       dev);
+		devl_health_reporter_create(devlink,
+					    &mlx5_fw_fatal_reporter_ops,
+					    grace_period,
+					    dev);
 	if (IS_ERR(health->fw_fatal_reporter))
 		mlx5_core_warn(dev, "Failed to create fw fatal reporter, err = %ld\n",
 			       PTR_ERR(health->fw_fatal_reporter));
@@ -777,7 +777,8 @@ void mlx5_trigger_health_work(struct mlx5_core_dev *dev)
 {
 	struct mlx5_core_health *health = &dev->priv.health;
 
-	queue_work(health->wq, &health->fatal_report_work);
+	if (!mlx5_dev_is_lightweight(dev))
+		queue_work(health->wq, &health->fatal_report_work);
 }
 
 #define MLX5_MSEC_PER_HOUR (MSEC_PER_SEC * 60 * 60)
@@ -905,10 +906,15 @@ void mlx5_health_cleanup(struct mlx5_core_dev *dev)
 
 int mlx5_health_init(struct mlx5_core_dev *dev)
 {
+	struct devlink *devlink = priv_to_devlink(dev);
 	struct mlx5_core_health *health;
 	char *name;
 
-	mlx5_fw_reporters_create(dev);
+	if (!mlx5_dev_is_lightweight(dev)) {
+		devl_lock(devlink);
+		mlx5_fw_reporters_create(dev);
+		devl_unlock(devlink);
+	}
 	mlx5_reporter_vnic_create(dev);
 
 	health = &dev->priv.health;
drivers/net/ethernet/mellanox/mlx5/core/main.c +116 -8

163 lines not on this page · raw message

drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h +7 -0
index 7a5f04082058..464c6885a01c 100644
@@ -240,11 +240,14 @@ int mlx5_attach_device(struct mlx5_core_dev *dev);
 void mlx5_detach_device(struct mlx5_core_dev *dev, bool suspend);
 int mlx5_register_device(struct mlx5_core_dev *dev);
 void mlx5_unregister_device(struct mlx5_core_dev *dev);
+void mlx5_dev_set_lightweight(struct mlx5_core_dev *dev);
+bool mlx5_dev_is_lightweight(struct mlx5_core_dev *dev);
 struct mlx5_core_dev *mlx5_get_next_phys_dev_lag(struct mlx5_core_dev *dev);
 void mlx5_dev_list_lock(void);
 void mlx5_dev_list_unlock(void);
 int mlx5_dev_list_trylock(void);
 
+void mlx5_fw_reporters_create(struct mlx5_core_dev *dev);
 int mlx5_query_mtpps(struct mlx5_core_dev *dev, u32 *mtpps, u32 mtpps_size);
 int mlx5_set_mtpps(struct mlx5_core_dev *mdev, u32 *mtpps, u32 mtpps_size);
 int mlx5_query_mtppse(struct mlx5_core_dev *mdev, u8 pin, u8 *arm, u8 *mode);
@@ -319,11 +322,15 @@ static inline bool mlx5_core_is_sf(const struct mlx5_core_dev *dev)
 int mlx5_mdev_init(struct mlx5_core_dev *dev, int profile_idx);
 void mlx5_mdev_uninit(struct mlx5_core_dev *dev);
 int mlx5_init_one(struct mlx5_core_dev *dev);
+int mlx5_init_one_devl_locked(struct mlx5_core_dev *dev);
 void mlx5_uninit_one(struct mlx5_core_dev *dev);
 void mlx5_unload_one(struct mlx5_core_dev *dev, bool suspend);
 void mlx5_unload_one_devl_locked(struct mlx5_core_dev *dev, bool suspend);
 int mlx5_load_one(struct mlx5_core_dev *dev, bool recovery);
 int mlx5_load_one_devl_locked(struct mlx5_core_dev *dev, bool recovery);
+int mlx5_init_one_light(struct mlx5_core_dev *dev);
+void mlx5_uninit_one_light(struct mlx5_core_dev *dev);
+void mlx5_unload_one_light(struct mlx5_core_dev *dev);
 
 int mlx5_vport_set_other_func_cap(struct mlx5_core_dev *dev, const void *hca_cap, u16 vport,
 				  u16 opmod);
drivers/net/ethernet/mellanox/mlx5/core/sf/dev/driver.c +13 -2
index 0692363cf80e..8fe82f1191bb 100644
@@ -3,6 +3,7 @@
 
 #include <linux/mlx5/driver.h>
 #include <linux/mlx5/device.h>
+#include <linux/mlx5/eswitch.h>
 #include "mlx5_core.h"
 #include "dev.h"
 #include "devlink.h"
@@ -28,6 +29,10 @@ static int mlx5_sf_dev_probe(struct auxiliary_device *adev, const struct auxilia
 	mdev->priv.adev_idx = adev->id;
 	sf_dev->mdev = mdev;
 
+	/* Only local SFs do light probe */
+	if (MLX5_ESWITCH_MANAGER(sf_dev->parent_mdev))
+		mlx5_dev_set_lightweight(mdev);
+
 	err = mlx5_mdev_init(mdev, MLX5_SF_PROF);
 	if (err) {
 		mlx5_core_warn(mdev, "mlx5_mdev_init on err=%d\n", err);
@@ -41,7 +46,10 @@ static int mlx5_sf_dev_probe(struct auxiliary_device *adev, const struct auxilia
 		goto remap_err;
 	}
 
-	err = mlx5_init_one(mdev);
+	if (MLX5_ESWITCH_MANAGER(sf_dev->parent_mdev))
+		err = mlx5_init_one_light(mdev);
+	else
+		err = mlx5_init_one(mdev);
 	if (err) {
 		mlx5_core_warn(mdev, "mlx5_init_one err=%d\n", err);
 		goto init_one_err;
@@ -65,7 +73,10 @@ static void mlx5_sf_dev_remove(struct auxiliary_device *adev)
 
 	mlx5_drain_health_wq(sf_dev->mdev);
 	devlink_unregister(devlink);
-	mlx5_uninit_one(sf_dev->mdev);
+	if (mlx5_dev_is_lightweight(sf_dev->mdev))
+		mlx5_uninit_one_light(sf_dev->mdev);
+	else
+		mlx5_uninit_one(sf_dev->mdev);
 	iounmap(sf_dev->mdev->iseg);
 	mlx5_mdev_uninit(sf_dev->mdev);
 	mlx5_devlink_free(devlink);
Signature · 1 line
2.40.1
maintainer
#16 lore raw

Re: [net-next 14/15] net/mlx5: Light probe local SFs

Quoted 7 lines · On Fri, 9 Jun 2023 18:42:53 -0700 Saeed Mahameed wrote:

In case user wants to configure the SFs, for example: to use only vdpa functionality, he needs to fully probe a SF, configure what he wants, and afterward reload the SF.

In order to save the time of the reload, local SFs will probe without any auxiliary sub-device, so that the SFs can be configured prior to its full probe.

I feel like we talked about this at least twice already, and I keep saying that the features should be specified when the device is spawned. Am I misremembering?

Will this patch not surprise existing users? You're changing the defaults. Does "local" mean on the IPU? Also "lightweight" feels uncomfortably close to marketing language.

Quoted 4 lines

The defaults of the enable_* devlink params of these SFs are set to false.

Usage example:

Is this a real example? Because we have..

Quoted 2 lines

Create SF:

$ devlink port add pci/0000:08:00.0 flavour pcisf pfnum 0 sfnum 11

sfnum 11 here

Quoted 1 line
$ devlink port function set pci/0000:08:00.0/32768 \

then port is 32768

Quoted 4 lines
               hw_addr 00:00:00:00:00:11 state active

Enable ETH auxiliary device:

$ devlink dev param set auxiliary/mlx5_core.sf.1 \

and instance is sf.1

Quoted 7 lines

name enable_eth value true cmode driverinit

Now, in order to fully probe the SF, use devlink reload:

$ devlink dev reload auxiliary/mlx5_core.sf.1

At this point the user have SF devlink instance with auxiliary device for the Ethernet functionality only.

maintainer
#17 lore raw

Re: [net-next 14/15] net/mlx5: Light probe local SFs

Quoted 5 lines · Jakub Kicinski
Quoted 4 lines · On Fri, 9 Jun 2023 18:42:53 -0700 Saeed Mahameed wrote:

In case user wants to configure the SFs, for example: to use only vdpa functionality, he needs to fully probe a SF, configure what he wants, and afterward reload the SF.

Quoted 8 lines
Quoted 3 lines

In order to save the time of the reload, local SFs will probe without any auxiliary sub-device, so that the SFs can be configured prior to its full probe.

I feel like we talked about this at least twice already, and I keep saying that the features should be specified when the device is spawned. Am I misremembering?

I think we did talk about this, but after internal research we prefer to
avoid adding additional knobs, unless you insist :) .. I think we already did a research and we feel that all of our users are going to re-configure the SF anyway, so why not make all SFs start with
"blank state" ?

Quoted 1 line

Will this patch not surprise existing users? You're changing the

I think we already checked, the feature is still not widely known. Let me double check.

Quoted 3 lines

defaults. Does "local" mean on the IPU? Also "lightweight" feels uncomfortably close to marketing language.

That wasn't out intention, poor choice of words, will reword to "blank SF"

Quoted 12 lines
Quoted 4 lines

The defaults of the enable_* devlink params of these SFs are set to false.

Usage example:

Is this a real example? Because we have..

Quoted 2 lines

Create SF:

$ devlink port add pci/0000:08:00.0 flavour pcisf pfnum 0 sfnum 11

sfnum 11 here

This an arbitrary user index.

Quoted 4 lines
Quoted 1 line
$ devlink port function set pci/0000:08:00.0/32768 \

then port is 32768

This is the actual HW port index, our SFs indexing start with an offset.

Quoted 7 lines
Quoted 4 lines
               hw_addr 00:00:00:00:00:11 state active

Enable ETH auxiliary device:

$ devlink dev param set auxiliary/mlx5_core.sf.1 \

and instance is sf.1

This was the first SF aux dev to be created on the system. :/

It's a mess ha...

Maybe we need to set the SF aux device index the same as the user index. But the HW/port index will always be different, otherwise we will need a map inside the driver.

#18 lore raw

Re: [net-next 14/15] net/mlx5: Light probe local SFs

Quoted 15 lines · Saeed Mahameed
Quoted 5 lines · Jakub Kicinski
Quoted 4 lines · On Fri,  9 Jun 2023 18:42:53 -0700 Saeed Mahameed wrote:

In case user wants to configure the SFs, for example: to use only vdpa functionality, he needs to fully probe a SF, configure what he wants, and afterward reload the SF.

Quoted 8 lines
Quoted 3 lines

In order to save the time of the reload, local SFs will probe without any auxiliary sub-device, so that the SFs can be configured prior to its full probe.

I feel like we talked about this at least twice already, and I keep saying that the features should be specified when the device is spawned. Am I misremembering?

Quoted 4 lines

I think we did talk about this, but after internal research we prefer to
avoid adding additional knobs, unless you insist :) .. I think we already did a research and we feel that all of our users are going to re-configure the SF anyway, so why not make all SFs start with
"blank state" ?

Shouldn't this be a devlink port param to enable/disable a specific feature on the SF before it is activated rather than making it a dev param on the SF aux device and requiring a devlink reload?

Quoted 42 lines
Quoted 2 lines

Will this patch not surprise existing users? You're changing the

I think we already checked, the feature is still not widely known. Let me double check.

Quoted 3 lines

defaults. Does "local" mean on the IPU? Also "lightweight" feels uncomfortably close to marketing language.

That wasn't out intention, poor choice of words, will reword to "blank SF"

Quoted 12 lines
Quoted 4 lines

The defaults of the enable_* devlink params of these SFs are set to false.

Usage example:

Is this a real example? Because we have..

Quoted 2 lines

Create SF:

$ devlink port add pci/0000:08:00.0 flavour pcisf pfnum 0 sfnum 11

sfnum 11 here

This an arbitrary user index.

Quoted 4 lines
Quoted 1 line
$ devlink port function set pci/0000:08:00.0/32768 \

then port is 32768

This is the actual HW port index, our SFs indexing start with an offset.

Quoted 7 lines
Quoted 4 lines

               hw_addr 00:00:00:00:00:11 state active

Enable ETH auxiliary device:

$ devlink dev param set auxiliary/mlx5_core.sf.1 \

and instance is sf.1

Quoted 7 lines

This was the first SF aux dev to be created on the system. :/

It's a mess ha...

Maybe we need to set the SF aux device index the same as the user index. But the HW/port index will always be different, otherwise we will need a map inside the driver.

Yes. Associating sfnum passed by user when creating a SF with the aux device would make it easier for orchestration tools to identify the aux device corresponding to a SF.

#19 lore raw

Re: [net-next 14/15] net/mlx5: Light probe local SFs

Quoted 18 lines · Sridhar
Quoted 15 lines · Saeed Mahameed
Quoted 13 lines · Jakub Kicinski
Quoted 7 lines · On Fri,  9 Jun 2023 18:42:53 -0700 Saeed Mahameed wrote:

In case user wants to configure the SFs, for example: to use only vdpa functionality, he needs to fully probe a SF, configure what he wants, and afterward reload the SF.

In order to save the time of the reload, local SFs will probe without any auxiliary sub-device, so that the SFs can be configured prior to its full probe.

I feel like we talked about this at least twice already, and I keep saying that the features should be specified when the device is spawned. Am I misremembering?

Quoted 7 lines
Quoted 4 lines

I think we did talk about this, but after internal research we prefer to
avoid adding additional knobs, unless you insist :) .. I think we already did a research and we feel that all of our users are going to re-configure the SF anyway, so why not make all SFs start with
"blank state" ?

Shouldn't this be a devlink port param to enable/disable a specific feature on the SF before it is activated rather than making it a dev param on the SF aux device and requiring a devlink reload?

Specific virtual HCA/SF/VF features are not directly related to what aux devices to expose, this is a separate feature that we are currently working on.

We are going to add devlink port params to have a granular control of what features the SF/VF will expose, e.g ipsec, tls, etc ..

These would affect the internal characteristic of the aux ULPs (netdev, rdma,vdpa) ..

But in this series, we improve the orchestration process of SF and what aux devs would spawn with it by default.. we already have an API to enable/disable what aux devs an SF would have, here we just improve the sequence of creating the SF.

Quoted 52 lines
Quoted 42 lines
Quoted 2 lines

Will this patch not surprise existing users? You're changing the

I think we already checked, the feature is still not widely known. Let me double check.

Quoted 3 lines

defaults. Does "local" mean on the IPU? Also "lightweight" feels uncomfortably close to marketing language.

That wasn't out intention, poor choice of words, will reword to "blank SF"

Quoted 12 lines
Quoted 4 lines

The defaults of the enable_* devlink params of these SFs are set to false.

Usage example:

Is this a real example? Because we have..

Quoted 2 lines

Create SF:

$ devlink port add pci/0000:08:00.0 flavour pcisf pfnum 0 sfnum 11

sfnum 11 here

This an arbitrary user index.

Quoted 4 lines
Quoted 1 line
$ devlink port function set pci/0000:08:00.0/32768 \

then port is 32768

This is the actual HW port index, our SFs indexing start with an offset.

Quoted 7 lines
Quoted 4 lines

               hw_addr 00:00:00:00:00:11 state active

Enable ETH auxiliary device:

$ devlink dev param set auxiliary/mlx5_core.sf.1 \

and instance is sf.1

Quoted 7 lines

This was the first SF aux dev to be created on the system. :/

It's a mess ha...

Maybe we need to set the SF aux device index the same as the user index. But the HW/port index will always be different, otherwise we will need a map inside the driver.

Yes. Associating sfnum passed by user when creating a SF with the aux device would make it easier for orchestration tools to identify the aux device corresponding to a SF.

maintainer
#20 lore raw

Re: [net-next 14/15] net/mlx5: Light probe local SFs

In reply to Saeed Mahameed
Quoted 14 lines · On Sat, 10 Jun 2023 21:15:57 -0700 Saeed Mahameed wrote:
Quoted 5 lines · Jakub Kicinski
Quoted 4 lines · On Fri, 9 Jun 2023 18:42:53 -0700 Saeed Mahameed wrote:

In case user wants to configure the SFs, for example: to use only vdpa functionality, he needs to fully probe a SF, configure what he wants, and afterward reload the SF.

Quoted 7 lines
Quoted 3 lines

In order to save the time of the reload, local SFs will probe without any auxiliary sub-device, so that the SFs can be configured prior to its full probe.

I feel like we talked about this at least twice already, and I keep saying that the features should be specified when the device is spawned. Am I misremembering?

Quoted 5 lines

I think we did talk about this, but after internal research we prefer to avoid adding additional knobs, unless you insist :) ..
I think we already did a research and we feel that all of our users are going to re-configure the SF anyway, so why not make all SFs start with "blank state" ?

In the container world, at least, I would have thought that the management daemon gets a full spec of the container its starting upfront. So going thru this spawn / config / futz / reset cycle is pure boilerplate pain.

What use cases are you considering? More VM-oriented?

Quoted 27 lines
Quoted 11 lines
Quoted 4 lines

The defaults of the enable_* devlink params of these SFs are set to false.

Usage example:

Is this a real example? Because we have..

Quoted 2 lines

Create SF:

$ devlink port add pci/0000:08:00.0 flavour pcisf pfnum 0 sfnum 11  

sfnum 11 here

This an arbitrary user index.

Quoted 3 lines
Quoted 1 line
$ devlink port function set pci/0000:08:00.0/32768 \  

then port is 32768

This is the actual HW port index, our SFs indexing start with an offset.

Quoted 6 lines
Quoted 4 lines
               hw_addr 00:00:00:00:00:11 state active

Enable ETH auxiliary device:

$ devlink dev param set auxiliary/mlx5_core.sf.1 \  

and instance is sf.1

Quoted 7 lines

This was the first SF aux dev to be created on the system. :/

It's a mess ha...

Maybe we need to set the SF aux device index the same as the user index. But the HW/port index will always be different, otherwise we will need a map inside the driver.

It'd be best to synchronously return to the user what the ID of the allocated entity is. It should be possible with some core changes to rig up devlink to return the sfnum and port ID. But IDK about the new devlink instance :(

#21 lore raw

Re: [net-next 14/15] net/mlx5: Light probe local SFs

Quoted 21 lines · Jakub Kicinski
Quoted 14 lines · On Sat, 10 Jun 2023 21:15:57 -0700 Saeed Mahameed wrote:
Quoted 5 lines · Jakub Kicinski
Quoted 4 lines · On Fri, 9 Jun 2023 18:42:53 -0700 Saeed Mahameed wrote:

In case user wants to configure the SFs, for example: to use only vdpa functionality, he needs to fully probe a SF, configure what he wants, and afterward reload the SF.

Quoted 7 lines
Quoted 3 lines

In order to save the time of the reload, local SFs will probe without any auxiliary sub-device, so that the SFs can be configured prior to its full probe.

I feel like we talked about this at least twice already, and I keep saying that the features should be specified when the device is spawned. Am I misremembering?

Quoted 5 lines

I think we did talk about this, but after internal research we prefer to avoid adding additional knobs, unless you insist :) ..
I think we already did a research and we feel that all of our users are going to re-configure the SF anyway, so why not make all SFs start with "blank state" ?

Quoted 5 lines

In the container world, at least, I would have thought that the management daemon gets a full spec of the container its starting upfront. So going thru this spawn / config / futz / reset cycle is pure boilerplate pain.

That's the point of the series. create / config / spawn.

personally I like that the SF object is created blank, with dev handles (devlink/aux) to configure it, and spawn it when ready.
I don't see a point of having an extra "blank state" devlink param.

Quoted 2 lines

What use cases are you considering? More VM-oriented?

Mostly container oriented, and selecting the ULP stacks, e.g RDMA, VDPA, virtio, netdev, etc ..

Quoted 31 lines
Quoted 31 lines
Quoted 11 lines
Quoted 4 lines

The defaults of the enable_* devlink params of these SFs are set to false.

Usage example:

Is this a real example? Because we have..

Quoted 2 lines

Create SF:

$ devlink port add pci/0000:08:00.0 flavour pcisf pfnum 0 sfnum 11

sfnum 11 here

This an arbitrary user index.

Quoted 3 lines
Quoted 1 line
$ devlink port function set pci/0000:08:00.0/32768 \

then port is 32768

This is the actual HW port index, our SFs indexing start with an offset.

Quoted 6 lines
Quoted 4 lines
               hw_addr 00:00:00:00:00:11 state active

Enable ETH auxiliary device:

$ devlink dev param set auxiliary/mlx5_core.sf.1 \

and instance is sf.1

This was the first SF aux dev to be created on the system. :/

It's a mess ha...

Quoted 8 lines
Quoted 3 lines

Maybe we need to set the SF aux device index the same as the user index. But the HW/port index will always be different, otherwise we will need a map inside the driver.

It'd be best to synchronously return to the user what the ID of the allocated entity is. It should be possible with some core changes to rig up devlink to return the sfnum and port ID. But IDK about the new devlink instance :(

I think that's possible, let me ask the team to take a shot at this..

I am not sure I understand what you mean by "new devlink instance".

SF creation will result in spawning two devlink handles, the SF function port of on the eswitch and the SF device devlink instance..

maintainer
#22 lore raw

Re: [net-next 14/15] net/mlx5: Light probe local SFs

Quoted 13 lines · On Tue, 13 Jun 2023 16:32:07 -0700 Saeed Mahameed wrote:
Quoted 11 lines · Jakub Kicinski
Quoted 5 lines · On Sat, 10 Jun 2023 21:15:57 -0700 Saeed Mahameed wrote:

I think we did talk about this, but after internal research we prefer to avoid adding additional knobs, unless you insist :) ..
I think we already did a research and we feel that all of our users are going to re-configure the SF anyway, so why not make all SFs start with "blank state" ?

In the container world, at least, I would have thought that the management daemon gets a full spec of the container its starting upfront. So going thru this spawn / config / futz / reset cycle is pure boilerplate pain.

Quoted 4 lines

That's the point of the series. create / config / spawn.

personally I like that the SF object is created blank, with dev handles (devlink/aux) to configure it, and spawn it when ready.

I think we had this discussion before, wasn't the initial proposal for SF along those lines? And we're slowly trending back towards ports in uninitialized state. It's okay, too late now.

Quoted 1 line

I don't see a point of having an extra "blank state" devlink param.

Yeah, the param would be worse of both worlds. We'll need to ensure consistency in other vendors, tho.

Quoted 4 lines
Quoted 1 line

What use cases are you considering? More VM-oriented?

Mostly container oriented, and selecting the ULP stacks, e.g RDMA, VDPA, virtio, netdev, etc ..

Odd, okay.

Quoted 13 lines
Quoted 4 lines
Quoted 4 lines

This was the first SF aux dev to be created on the system. :/

It's a mess ha...

Quoted 8 lines
Quoted 3 lines

Maybe we need to set the SF aux device index the same as the user index. But the HW/port index will always be different, otherwise we will need a map inside the driver.

It'd be best to synchronously return to the user what the ID of the allocated entity is. It should be possible with some core changes to rig up devlink to return the sfnum and port ID. But IDK about the new devlink instance :(

Quoted 6 lines

I think that's possible, let me ask the team to take a shot at this..

I am not sure I understand what you mean by "new devlink instance".

SF creation will result in spawning two devlink handles, the SF function port of on the eswitch and the SF device devlink instance..

Yes, I mean "SF device devlink instance" by "new devlink instance".

In theory this should all be doable with netlink. NLM_F_ECHO should loop all notifications back to the requester. The tricky part is catching the notifications, I'm guessing, because in theory the devlink instance spawning may be async for locking reasons? Hopefully not, then it's easy..

#23 lore raw

Re: [net-next 14/15] net/mlx5: Light probe local SFs

Quoted 55 lines · Wed, Jun 14, 2023 at 04:05:52AM CEST, [email protected] wrote:
Quoted 13 lines · On Tue, 13 Jun 2023 16:32:07 -0700 Saeed Mahameed wrote:
Quoted 11 lines · Jakub Kicinski
Quoted 5 lines · On Sat, 10 Jun 2023 21:15:57 -0700 Saeed Mahameed wrote:

I think we did talk about this, but after internal research we prefer to avoid adding additional knobs, unless you insist :) ..
I think we already did a research and we feel that all of our users are going to re-configure the SF anyway, so why not make all SFs start with "blank state" ?

In the container world, at least, I would have thought that the management daemon gets a full spec of the container its starting upfront. So going thru this spawn / config / futz / reset cycle is pure boilerplate pain.

Quoted 4 lines

That's the point of the series. create / config / spawn.

personally I like that the SF object is created blank, with dev handles (devlink/aux) to configure it, and spawn it when ready.

I think we had this discussion before, wasn't the initial proposal for SF along those lines? And we're slowly trending back towards ports in uninitialized state. It's okay, too late now.

Quoted 1 line

I don't see a point of having an extra "blank state" devlink param.

Yeah, the param would be worse of both worlds. We'll need to ensure consistency in other vendors, tho.

Quoted 4 lines
Quoted 1 line

What use cases are you considering? More VM-oriented?

Mostly container oriented, and selecting the ULP stacks, e.g RDMA, VDPA, virtio, netdev, etc ..

Odd, okay.

Quoted 13 lines
Quoted 4 lines
Quoted 4 lines

This was the first SF aux dev to be created on the system. :/

It's a mess ha...

Quoted 8 lines
Quoted 3 lines

Maybe we need to set the SF aux device index the same as the user index. But the HW/port index will always be different, otherwise we will need a map inside the driver.

It'd be best to synchronously return to the user what the ID of the allocated entity is. It should be possible with some core changes to rig up devlink to return the sfnum and port ID. But IDK about the new devlink instance :(

Quoted 6 lines

I think that's possible, let me ask the team to take a shot at this..

I am not sure I understand what you mean by "new devlink instance".

SF creation will result in spawning two devlink handles, the SF function port of on the eswitch and the SF device devlink instance..

Quoted 7 lines

Yes, I mean "SF device devlink instance" by "new devlink instance".

In theory this should all be doable with netlink. NLM_F_ECHO should loop all notifications back to the requester. The tricky part is catching the notifications, I'm guessing, because in theory the devlink instance spawning may be async for locking reasons? Hopefully not, then it's easy..

The problem is scalability. SFs could be activated in parallel, but the cmd that is doing that holds devlink instance lock. That serializes it. So we need to either:
1) change the devlink locking to be able to execute some of the cmds in parallel and leave the activation sync
2) change the activation to be async and work with notifications

I like 2) better, as the 1) maze we just got out of recently :) WDYT?

Quoted 1 line
maintainer
#24 lore raw

Re: [net-next 14/15] net/mlx5: Light probe local SFs

Quoted 9 lines · On Thu, 15 Jun 2023 12:51:09 +0200 Jiri Pirko wrote:

The problem is scalability. SFs could be activated in parallel, but the cmd that is doing that holds devlink instance lock. That serializes it. So we need to either:
1) change the devlink locking to be able to execute some of the cmds in parallel and leave the activation sync
2) change the activation to be async and work with notifications

I like 2) better, as the 1) maze we just got out of recently :) WDYT?

I guess we don't need to wait for the full activation. Is the port creation also async, then, or just the SF devlink instance creation?

#25 lore raw

Re: [net-next 14/15] net/mlx5: Light probe local SFs

Quoted 8 lines · Thu, Jun 15, 2023 at 06:37:01PM CEST, [email protected] wrote:
Quoted 7 lines · On Thu, 15 Jun 2023 12:51:09 +0200 Jiri Pirko wrote:

The problem is scalability. SFs could be activated in parallel, but the cmd that is doing that holds devlink instance lock. That serializes it. So we need to either:
1) change the devlink locking to be able to execute some of the cmds in parallel and leave the activation sync
2) change the activation to be async and work with notifications

Quoted 5 lines
Quoted 2 lines

I like 2) better, as the 1) maze we just got out of recently :) WDYT?

I guess we don't need to wait for the full activation. Is the port creation also async, then, or just the SF devlink instance creation?

I'm not sure I follow :/
The activation is when the SF auxiliary device is created. The driver then probes the SF auxiliary device and instantiates everything, SF devlink, SF netdev, etc.

We need wait/notification for 2 reasons
1) to get the auxiliary device name for the activated SF. It is needed for convenience of the orchestration tools.
2) to get the result of the activation (success/fail) It is also needed for convenience of the orchestration tools.

maintainer
#26 lore raw

Re: [net-next 14/15] net/mlx5: Light probe local SFs

Quoted 12 lines · On Thu, 15 Jun 2023 19:37:23 +0200 Jiri Pirko wrote:
Quoted 11 lines · Thu, Jun 15, 2023 at 06:37:01PM CEST, [email protected] wrote:
Quoted 9 lines · On Thu, 15 Jun 2023 12:51:09 +0200 Jiri Pirko wrote:

The problem is scalability. SFs could be activated in parallel, but the cmd that is doing that holds devlink instance lock. That serializes it. So we need to either:
1) change the devlink locking to be able to execute some of the cmds in parallel and leave the activation sync
2) change the activation to be async and work with notifications

I like 2) better, as the 1) maze we just got out of recently :) WDYT?

Quoted 7 lines
Quoted 2 lines

I guess we don't need to wait for the full activation. Is the port creation also async, then, or just the SF devlink instance creation?

I'm not sure I follow :/
The activation is when the SF auxiliary device is created. The driver then probes the SF auxiliary device and instantiates everything, SF devlink, SF netdev, etc.

Sorry, maybe let's look at an example:

$ devlink port add pci/0000:08:00.0 flavour pcisf pfnum 0 sfnum 11

needs to print / return the handle of the created port.

$ devlink port function set pci/0000:08:00.0/32768 \
               hw_addr 00:00:00:00:00:11 state active

needs to print / return the handle of the devlink instance

Quoted 5 lines

We need wait/notification for 2 reasons
1) to get the auxiliary device name for the activated SF. It is needed for convenience of the orchestration tools.
2) to get the result of the activation (success/fail) It is also needed for convenience of the orchestration tools.

Are you saying the activation already waits for the devlink instance to be spawned? If so that's great, all we need to do is for the:

$ devlink port function set pci/0000:08:00.0/32768 \
               hw_addr 00:00:00:00:00:11 state active

to either return sufficient info for the orchestration to know what the resulting SF / SF devlink instance is. Most likely indirectly by adding that info to the port so that the PORT_NEW notification carries it.

Did I confuse things even more?

As a reminder what sparked this convo is that user specifies "sfnum 11" in the example, and the sf device gets called "sf.1".

#27 lore raw

Re: [net-next 14/15] net/mlx5: Light probe local SFs

Quoted 21 lines · Thu, Jun 15, 2023 at 09:33:25PM CEST, [email protected] wrote:
Quoted 12 lines · On Thu, 15 Jun 2023 19:37:23 +0200 Jiri Pirko wrote:
Quoted 11 lines · Thu, Jun 15, 2023 at 06:37:01PM CEST, [email protected] wrote:
Quoted 9 lines · On Thu, 15 Jun 2023 12:51:09 +0200 Jiri Pirko wrote:

The problem is scalability. SFs could be activated in parallel, but the cmd that is doing that holds devlink instance lock. That serializes it. So we need to either:
1) change the devlink locking to be able to execute some of the cmds in parallel and leave the activation sync
2) change the activation to be async and work with notifications

I like 2) better, as the 1) maze we just got out of recently :) WDYT?

Quoted 7 lines
Quoted 2 lines

I guess we don't need to wait for the full activation. Is the port creation also async, then, or just the SF devlink instance creation?

I'm not sure I follow :/
The activation is when the SF auxiliary device is created. The driver then probes the SF auxiliary device and instantiates everything, SF devlink, SF netdev, etc.

Quoted 5 lines

Sorry, maybe let's look at an example:

$ devlink port add pci/0000:08:00.0 flavour pcisf pfnum 0 sfnum 11

needs to print / return the handle of the created port.

Yep, that is what happens now.

Quoted 5 lines
$ devlink port function set pci/0000:08:00.0/32768 \
              hw_addr 00:00:00:00:00:11 state active

needs to print / return the handle of the devlink instance

Right, that makes sense. I will look into adding that. It's a bit tricky, as the instance might actually appear on a different host, then, there is nothing to return here.

Quoted 9 lines
Quoted 6 lines

We need wait/notification for 2 reasons
1) to get the auxiliary device name for the activated SF. It is needed for convenience of the orchestration tools.
2) to get the result of the activation (success/fail) It is also needed for convenience of the orchestration tools.

Are you saying the activation already waits for the devlink instance to be spawned? If so that's great, all we need to do is for the:

Yep.

Quoted 7 lines
$ devlink port function set pci/0000:08:00.0/32768 \
              hw_addr 00:00:00:00:00:11 state active

to either return sufficient info for the orchestration to know what the resulting SF / SF devlink instance is. Most likely indirectly by adding that info to the port so that the PORT_NEW notification carries it.

Yeah, we need to add the same info to the GET cmd.

Quoted 2 lines

Did I confuse things even more?

No, no confusion. But, the problem with this is that devlink port function set active blocks until the activation is done holding the devlink instance lock. That prevents other ports from being activated in parallel. From driver/FW/HW perspective, we can do that.

So the question is, how to allow this parallelism?

Quoted 3 lines

As a reminder what sparked this convo is that user specifies "sfnum 11" in the example, and the sf device gets called "sf.1".

Yeah, will look into that as well.

maintainer
#28 lore raw

Re: [net-next 14/15] net/mlx5: Light probe local SFs

Quoted 8 lines · On Wed, 21 Jun 2023 15:14:35 +0200 Jiri Pirko wrote:
Quoted 1 line

Did I confuse things even more?

No, no confusion. But, the problem with this is that devlink port function set active blocks until the activation is done holding the devlink instance lock. That prevents other ports from being activated in parallel. From driver/FW/HW perspective, we can do that.

So the question is, how to allow this parallelism?

You seem to be concerned about parallelism, maybe you can share some details / data / calculations? I don't think that we need to hold the instance lock just to get the notification but I'd strongly prefer not to complicate things until problem actually exists.

The recent problems in the rtnl-lock-less flower implementation made me very cautious about complicating the stack because someone's FW is slow.

#29 lore raw

Re: [net-next 14/15] net/mlx5: Light probe local SFs

Quoted 10 lines · Wed, Jun 21, 2023 at 08:23:53PM CEST, [email protected] wrote:
Quoted 8 lines · On Wed, 21 Jun 2023 15:14:35 +0200 Jiri Pirko wrote:
Quoted 1 line

Did I confuse things even more?

No, no confusion. But, the problem with this is that devlink port function set active blocks until the activation is done holding the devlink instance lock. That prevents other ports from being activated in parallel. From driver/FW/HW perspective, we can do that.

So the question is, how to allow this parallelism?

Quoted 7 lines

You seem to be concerned about parallelism, maybe you can share some details / data / calculations? I don't think that we need to hold the instance lock just to get the notification but I'd strongly prefer not to complicate things until problem actually exists.

The recent problems in the rtnl-lock-less flower implementation made me very cautious about complicating the stack because someone's FW is slow.

I agree 100%. That is my stand as well. Lock less devlink commands are a simple no-go from my perspective. We got out of that mess only recently.

I just checked mlx5 code, the SF activation does not block. It just triggers activation of SF in FW and returns. Does not wait for completion.

Let me figure out the devlink_port<->sf_devlink linkage and I believe that would be enough for all needed usecases.

#30 lore raw

Re: [net-next 14/15] net/mlx5: Light probe local SFs

In reply to Jiri Pirko
Quoted 4 lines · Wed, Jun 21, 2023 at 03:14:35PM CEST, [email protected] wrote:
Quoted 3 lines · Thu, Jun 15, 2023 at 09:33:25PM CEST, [email protected] wrote:
Quoted 2 lines · On Thu, 15 Jun 2023 19:37:23 +0200 Jiri Pirko wrote:
Quoted 1 line · Thu, Jun 15, 2023 at 06:37:01PM CEST, [email protected] wrote:

On Thu, 15 Jun 2023 12:51:09 +0200 Jiri Pirko wrote:

[...]

Quoted 4 lines
Quoted 2 lines

As a reminder what sparked this convo is that user specifies "sfnum 11" in the example, and the sf device gets called "sf.1".

Yeah, will look into that as well.

I checked, the misalignment between sfnum and auxdev index. The problem is that the index space of sfnum is per-devlink instance, however the index space of auxdev is per module name.
So if you have one devlink instance managing eswitch, in theory we can map sfnum to auxdev id 1:1. But if you plug-in another physical nic, second devlink instance managing eswitch appears, then we have an overlap. I don't see any way out of this, do you?

But, I believe if we add a proper reference between devlink sf port and the actual sf devlink instace, that would be enough.

maintainer
#31 lore raw

Re: [net-next 14/15] net/mlx5: Light probe local SFs

Quoted 10 lines · On Thu, 22 Jun 2023 08:38:05 +0200 Jiri Pirko wrote:

I checked, the misalignment between sfnum and auxdev index. The problem is that the index space of sfnum is per-devlink instance, however the index space of auxdev is per module name.
So if you have one devlink instance managing eswitch, in theory we can map sfnum to auxdev id 1:1. But if you plug-in another physical nic, second devlink instance managing eswitch appears, then we have an overlap. I don't see any way out of this, do you?

But, I believe if we add a proper reference between devlink sf port and the actual sf devlink instace, that would be enough.

SG. For the IPU case when spawning from within the IPU can we still figure out what the auxdev id is going to be? If not maybe we should add some form of UUID on both the port and the sf devlink instance? Maybe there's already some UUID in the vfio world we can co-opt?

#32 lore raw

Re: [net-next 14/15] net/mlx5: Light probe local SFs

Quoted 9 lines · Thu, Jun 22, 2023 at 06:35:23PM CEST, [email protected] wrote:
Quoted 8 lines · On Thu, 22 Jun 2023 08:38:05 +0200 Jiri Pirko wrote:

I checked, the misalignment between sfnum and auxdev index. The problem is that the index space of sfnum is per-devlink instance, however the index space of auxdev is per module name.
So if you have one devlink instance managing eswitch, in theory we can map sfnum to auxdev id 1:1. But if you plug-in another physical nic, second devlink instance managing eswitch appears, then we have an overlap. I don't see any way out of this, do you?

Quoted 5 lines
Quoted 2 lines

But, I believe if we add a proper reference between devlink sf port and the actual sf devlink instace, that would be enough.

SG. For the IPU case when spawning from within the IPU can we still figure out what the auxdev id is going to be? If not maybe we should

Yeah, the driver is assigning the auxdev id. I'm now trying to figure out how to pass that to devlink code/user nicely. The devlink instance for the SF does not exist yet, but we know what the handle is going to be. Perhaps some sort of place holder instance would do. IDK.

Quoted 1 line

add some form of UUID on both the port and the sf devlink instance?

What about the MAC?

$ sudo devlink port add pci/0000:08:00.0 flavour pcisf pfnum 0 sfnum 102
pci/0000:08:00.0/32769: type eth netdev eth8 flavour pcisf controller 0 pfnum 0 sfnum 102 splittable false
  function:
    hw_addr 00:00:00:00:00:00 state inactive opstate detached
$ sudo devlink port function set pci/0000:08:00.0/32769 hw_addr AA:22:33:44:55:66
$ sudo devlink port function set pci/0000:08:00.0/32769 state active
$ ip link show eth9
15: eth9: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether aa:22:33:44:55:66 brd ff:ff:ff:ff:ff:ff

There are 2 issues with this:
1) If the hw_addr stays zeroed for activation, random MAC is generated
2) On the SF side, the MAC is only seen for netdev. That is problematic for SFs without netdev. However, I don't see why we cannot add devlink port attr to expose hw_addr on the SF.

Quoted 1 line

Maybe there's already some UUID in the vfio world we can co-opt?

Let me check that out.

maintainer
#33 lore raw

Re: [net-next 14/15] net/mlx5: Light probe local SFs

Quoted 23 lines · On Fri, 23 Jun 2023 11:27:10 +0200 Jiri Pirko wrote:
Quoted 2 lines · Thu, Jun 22, 2023 at 06:35:23PM CEST, [email protected] wrote:

SG. For the IPU case when spawning from within the IPU can we still figure out what the auxdev id is going to be? If not maybe we should

Yeah, the driver is assigning the auxdev id. I'm now trying to figure out how to pass that to devlink code/user nicely. The devlink instance for the SF does not exist yet, but we know what the handle is going to be. Perhaps some sort of place holder instance would do. IDK.

Quoted 1 line

add some form of UUID on both the port and the sf devlink instance?

What about the MAC?

$ sudo devlink port add pci/0000:08:00.0 flavour pcisf pfnum 0 sfnum 102
pci/0000:08:00.0/32769: type eth netdev eth8 flavour pcisf controller 0 pfnum 0 sfnum 102 splittable false
  function:
    hw_addr 00:00:00:00:00:00 state inactive opstate detached
$ sudo devlink port function set pci/0000:08:00.0/32769 hw_addr AA:22:33:44:55:66
$ sudo devlink port function set pci/0000:08:00.0/32769 state active
$ ip link show eth9
15: eth9: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether aa:22:33:44:55:66 brd ff:ff:ff:ff:ff:ff
Quoted 5 lines

There are 2 issues with this:
1) If the hw_addr stays zeroed for activation, random MAC is generated
2) On the SF side, the MAC is only seen for netdev. That is problematic for SFs without netdev. However, I don't see why we cannot add devlink port attr to expose hw_addr on the SF.

Yeah, the fact that mac add of zero has special meaning could be problematic. Other vendors may get "inspired" by the legacy SR-IOV semantics of MAC addr of zero == user can decide, or whatnot. The value on the port is the admin-set MAC, not the current MAC of the SF after all, right? Probably best to find another way...

Quoted 3 lines
Quoted 1 line

Maybe there's already some UUID in the vfio world we can co-opt?

Let me check that out.

#34 lore raw

Re: [net-next 14/15] net/mlx5: Light probe local SFs

Quoted 24 lines · Fri, Jun 23, 2023 at 05:21:08PM CEST, [email protected] wrote:
Quoted 23 lines · On Fri, 23 Jun 2023 11:27:10 +0200 Jiri Pirko wrote:
Quoted 2 lines · Thu, Jun 22, 2023 at 06:35:23PM CEST, [email protected] wrote:

SG. For the IPU case when spawning from within the IPU can we still figure out what the auxdev id is going to be? If not maybe we should

Yeah, the driver is assigning the auxdev id. I'm now trying to figure out how to pass that to devlink code/user nicely. The devlink instance for the SF does not exist yet, but we know what the handle is going to be. Perhaps some sort of place holder instance would do. IDK.

Quoted 1 line

add some form of UUID on both the port and the sf devlink instance?

What about the MAC?

$ sudo devlink port add pci/0000:08:00.0 flavour pcisf pfnum 0 sfnum 102
pci/0000:08:00.0/32769: type eth netdev eth8 flavour pcisf controller 0 pfnum 0 sfnum 102 splittable false
  function:
    hw_addr 00:00:00:00:00:00 state inactive opstate detached
$ sudo devlink port function set pci/0000:08:00.0/32769 hw_addr AA:22:33:44:55:66
$ sudo devlink port function set pci/0000:08:00.0/32769 state active
$ ip link show eth9
15: eth9: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether aa:22:33:44:55:66 brd ff:ff:ff:ff:ff:ff
Quoted 7 lines
Quoted 5 lines

There are 2 issues with this:
1) If the hw_addr stays zeroed for activation, random MAC is generated
2) On the SF side, the MAC is only seen for netdev. That is problematic for SFs without netdev. However, I don't see why we cannot add devlink port attr to expose hw_addr on the SF.

Yeah, the fact that mac add of zero has special meaning could be

True. But I believe we can sanitize this in devlink core, not allowing to activate zeroed mac. Instead of radom generating of a mac on the SF side, we random generate it on the eswitch port function side.

I think this can work.

Quoted 3 lines

problematic. Other vendors may get "inspired" by the legacy SR-IOV semantics of MAC addr of zero == user can decide, or whatnot. The value on the port is the admin-set MAC, not the current MAC

True, that is why I suggested to expose hw_addr attr on devlink port of the SF instance, where this would be fixed no matter what the user does on the actual netdevice.

Quoted 1 line

of the SF after all, right? Probably best to find another way...

Well, yeah. The mac/hw_addr is quite convenient. It's there and I believe that any device could work with that. Having some kind of "extra cookie" would require to implement that in FW, which makes things more complicated.

Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

Maybe there's already some UUID in the vfio world we can co-opt?

Let me check that out.

maintainer
#35 lore raw

Re: [net-next 14/15] net/mlx5: Light probe local SFs

Quoted 6 lines · On Sat, 24 Jun 2023 11:33:20 +0200 Jiri Pirko wrote:
Quoted 1 line

of the SF after all, right? Probably best to find another way...

Well, yeah. The mac/hw_addr is quite convenient. It's there and I believe that any device could work with that. Having some kind of "extra cookie" would require to implement that in FW, which makes things more complicated.

"Let's piggyback on something else in the uAPI because I don't want to extend FW" is not an argument which can be taken seriously.

#36 lore raw

Re: [net-next 14/15] net/mlx5: Light probe local SFs

Quoted 10 lines · Sat, Jun 24, 2023 at 10:47:03PM CEST, [email protected] wrote:
Quoted 6 lines · On Sat, 24 Jun 2023 11:33:20 +0200 Jiri Pirko wrote:
Quoted 1 line

of the SF after all, right? Probably best to find another way...

Well, yeah. The mac/hw_addr is quite convenient. It's there and I believe that any device could work with that. Having some kind of "extra cookie" would require to implement that in FW, which makes things more complicated.

"Let's piggyback on something else in the uAPI because I don't want to extend FW" is not an argument which can be taken seriously.

It's not about what I want or not. It's about what can realistically happen (not talking about mlx5 specifically).

But what is a difference between using hw_addr and some other uuid. cmdline wise:

option 1 - JUST HW_ADDR:

$ sudo devlink port add pci/0000:08:00.0 flavour pcisf pfnum 0 sfnum 103
pci/0000:08:00.0/32770: type eth netdev eth10 flavour pcisf controller 0 pfnum 0 sfnum 103 splittable false
  function:
    hw_addr 00:00:00:00:00:00 state inactive opstate detached
$ sudo devlink port function set pci/0000:08:00.0/32770 hw_addr AA:22:33:44:55:66
$ sudo devlink port show pci/0000:08:00.0/32770
pci/0000:08:00.0/32770: type eth netdev eth10 flavour pcisf controller 0 pfnum 0 sfnum 103 splittable false
  function:
    hw_addr aa:22:33:44:55:66 state inactive opstate detached
$ sudo devlink port function set pci/0000:08:00.0/32770 state active
pci/0000:08:00.0/32770: type eth netdev eth10 flavour pcisf controller 0 pfnum 0 sfnum 103 splittable false
  function:
    hw_addr aa:22:33:44:55:66 state active opstate detached devlink_handle auxiliary/mlx5_core.eth.5
# This is new, currently activation does not give feedback,
# devlink_handle attr is new here
$ sudo devlink port show pci/0000:08:00.0/32770
pci/0000:08:00.0/32770: type eth netdev eth10 flavour pcisf controller 0 pfnum 0 sfnum 103 splittable false
  function:
    hw_addr aa:22:33:44:55:66 state active opstate attached devlink_handle auxiliary/mlx5_core.eth.5
# devlink_handle attr is new here
$ sudo devlink dev show auxiliary/mlx5_core.eth.5
auxiliary/mlx5_core.eth.5: hw_addr aa:22:33:44:55:66
# hw_addr attr is new here

hw_addr value is passed through FW here. This value is currently used for netdevice mac. So it is just a matter of exposing for auxdev devlink/devlink_port (not 100% sure which)

option 2 - ADDED UUID:

$ sudo devlink port add pci/0000:08:00.0 flavour pcisf pfnum 0 sfnum 103
pci/0000:08:00.0/32770: type eth netdev eth10 flavour pcisf controller 0 pfnum 0 sfnum 103 splittable false
  function:
    hw_addr 00:00:00:00:00:00 state inactive opstate detached
$ sudo devlink port function set pci/0000:08:00.0/32770 hw_addr AA:22:33:44:55:66 uuid SOMETHINGREALLYUNIQUE
$ sudo devlink port show pci/0000:08:00.0/32770
pci/0000:08:00.0/32770: type eth netdev eth10 flavour pcisf controller 0 pfnum 0 sfnum 103 splittable false
  function:
    hw_addr aa:22:33:44:55:66 uuid SOMETHINGREALLYUNIQUE state inactive opstate detached
# uuid attr is new here
$ sudo devlink port function set pci/0000:08:00.0/32770 state active
pci/0000:08:00.0/32770: type eth netdev eth10 flavour pcisf controller 0 pfnum 0 sfnum 103 splittable false
  function:
    hw_addr aa:22:33:44:55:66 uuid SOMETHINGREALLYUNIQUE state active opstate detached devlink_handle auxiliary/mlx5_core.eth.5
# This is new, currently activation does not give feedback,
# devlink_handle attr is new here, uuid attr is new here
$ sudo devlink port show pci/0000:08:00.0/32770
pci/0000:08:00.0/32770: type eth netdev eth10 flavour pcisf controller 0 pfnum 0 sfnum 103 splittable false
  function:
    hw_addr aa:22:33:44:55:66 uuid SOMETHINGREALLYUNIQUE state active opstate attached devlink_handle auxiliary/mlx5_core.eth.5
# devlink_handle attr is new here, uuid attr is new here
$ sudo devlink dev show auxiliary/mlx5_core.eth.5
auxiliary/mlx5_core.eth.5: uuid SOMETHINGREALLYUNIQUE
# uuid attr is new here

uuid value is passed through FW here.

The uuid kind of values is nothing new in netdev. We have:
IFLA_PHYS_PORT_ID
IFLA_PHYS_SWITCH_ID
In most of the cases (if not all), these are in drivers filled-up with MAC. So I'm quite positive that the drivers would implement the auxdev uuid as a MAC as well.

Do you like option 2 better than option 1? Is there option 3?

Thanks!

maintainer
#37 lore raw

Re: [net-next 14/15] net/mlx5: Light probe local SFs

Quoted 6 lines · On Tue, 27 Jun 2023 12:12:10 +0200 Jiri Pirko wrote:
$ sudo devlink port add pci/0000:08:00.0 flavour pcisf pfnum 0 sfnum 103
pci/0000:08:00.0/32770: type eth netdev eth10 flavour pcisf controller 0 pfnum 0 sfnum 103 splittable false
  function:
    hw_addr 00:00:00:00:00:00 state inactive opstate detached
$ sudo devlink port function set pci/0000:08:00.0/32770 hw_addr AA:22:33:44:55:66 uuid SOMETHINGREALLYUNIQUE

Why does the user have to set the uuid? I was expecting it'd pop up on the port automatically, generated by the kernel, as a read-only attribute.

#38 lore raw

Re: [net-next 14/15] net/mlx5: Light probe local SFs

Quoted 11 lines · Tue, Jun 27, 2023 at 05:24:29PM CEST, [email protected] wrote:
Quoted 6 lines · On Tue, 27 Jun 2023 12:12:10 +0200 Jiri Pirko wrote:
$ sudo devlink port add pci/0000:08:00.0 flavour pcisf pfnum 0 sfnum 103
pci/0000:08:00.0/32770: type eth netdev eth10 flavour pcisf controller 0 pfnum 0 sfnum 103 splittable false
  function:
    hw_addr 00:00:00:00:00:00 state inactive opstate detached
$ sudo devlink port function set pci/0000:08:00.0/32770 hw_addr AA:22:33:44:55:66 uuid SOMETHINGREALLYUNIQUE

Why does the user have to set the uuid? I was expecting it'd pop up on the port automatically, generated by the kernel, as a read-only attribute.

Hmm, how that could be generated by kernel if it should be really unique? Consider an example scenario where you have 2 DPUS (smartnic with CPUs) plugged into a single host.

1) On DPU 1 you do:

   $ sudo devlink port add pci/0000:08:00.0 flavour pcisf pfnum 1 sfnum 103
   pci/0000:08:00.0/32770: type eth netdev eth10 flavour pcisf controller 1 pfnum 0 sfnum 103 splittable false
       function:
       hw_addr 00:00:00:00:00:00 state inactive opstate detached uuid XXX

2) On DPU 2 you do:

   $ sudo devlink port add pci/0000:08:00.0 flavour pcisf pfnum 1 sfnum 103
   pci/0000:08:00.0/32770: type eth netdev eth10 flavour pcisf controller 1 pfnum 0 sfnum 103 splittable false
       function:
       hw_addr 00:00:00:00:00:00 state inactive opstate detached uuid XXX

There is no way to sync between kernel running in the DPUs. Both SFs in this example would be externaly created on the host. The host will see 2 devices with the same uuid XXX, collision.

maintainer
#39 lore raw

Re: [net-next 14/15] net/mlx5: Light probe local SFs

Quoted 11 lines · On Tue, 27 Jun 2023 19:16:42 +0200 Jiri Pirko wrote:

Hmm, how that could be generated by kernel if it should be really unique? Consider an example scenario where you have 2 DPUS (smartnic with CPUs) plugged into a single host.

1) On DPU 1 you do:

   $ sudo devlink port add pci/0000:08:00.0 flavour pcisf pfnum 1 sfnum 103
   pci/0000:08:00.0/32770: type eth netdev eth10 flavour pcisf controller 1 pfnum 0 sfnum 103 splittable false
       function:
       hw_addr 00:00:00:00:00:00 state inactive opstate detached uuid XXX

2) On DPU 2 you do:

Quoted 8 lines
   $ sudo devlink port add pci/0000:08:00.0 flavour pcisf pfnum 1 sfnum 103
   pci/0000:08:00.0/32770: type eth netdev eth10 flavour pcisf controller 1 pfnum 0 sfnum 103 splittable false
       function:
       hw_addr 00:00:00:00:00:00 state inactive opstate detached uuid XXX

There is no way to sync between kernel running in the DPUs. Both SFs in this example would be externaly created on the host. The host will see 2 devices with the same uuid XXX, collision.

https://en.wikipedia.org/wiki/Universally_unique_identifier

tl;dr the whole point of UUIDs is that collisions do not happen

#40 lore raw

PATCH 15/15 net/mlx5e: Remove a useless function call

In reply to Saeed Mahameed

From: Christophe JAILLET <[email protected]>

'handle' is known to be NULL here. There is no need to kfree() it.

---
 drivers/net/ethernet/mellanox/mlx5/core/en/tc/post_act.c | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en/tc/post_act.c +1 -3
index 0290e0dea539..4e923a2874ae 100644
@@ -112,10 +112,8 @@ mlx5e_tc_post_act_add(struct mlx5e_post_act *post_act, struct mlx5_flow_attr *po
 	int err;
 
 	handle = kzalloc(sizeof(*handle), GFP_KERNEL);
-	if (!handle) {
-		kfree(handle);
+	if (!handle)
 		return ERR_PTR(-ENOMEM);
-	}
 
 	post_attr->chain = 0;
 	post_attr->prio = 0;
Signature · 1 line
2.40.1