Skip to contentSkip to conversation

mlxsw: Implement dev info and dev flash for line cards

PATCH In review Jiri Pirko 4 participants 32 messages Jun 14, 2022 – Jun 28, 2022

Subsystems (6)
lore
Share & tools
lore

Jiri Pirko

author maintainer
#0 lore raw

cover mlxsw: Implement dev info and dev flash for line cards

From: Jiri Pirko <[email protected]>

This patchset implements two features:
1) "devlink dev info" is exposed for line card (patches 3-8)
2) "devlink dev flash" is implemented for line card gearbox flashing (patch 9)

For every line card, "a nested" auxiliary device is created which allows to bind the features mentioned above (patch 2).

The relationship between line card and its auxiliary dev devlink is carried over extra line card netlink attribute (patches 1 and 3).

Examples:

$ devlink lc show pci/0000:01:00.0 lc 1
pci/0000:01:00.0:
  lc 1 state active type 16x100G nested_devlink auxiliary/mlxsw_core.lc.0
    supported_types:
       16x100G
$ devlink dev show auxiliary/mlxsw_core.lc.0
auxiliary/mlxsw_core.lc.0
$ devlink dev info auxiliary/mlxsw_core.lc.0
auxiliary/mlxsw_core.lc.0:
  versions:
      fixed:
        hw.revision 0
        fw.psid MT_0000000749
      running:
        ini.version 4
        fw 19.2010.1312
$ devlink dev flash auxiliary/mlxsw_core.lc.0 file mellanox/fw-AGB-rel-19_2010_1312-022-EVB.mfa2

Jiri Pirko (11):
devlink: introduce nested devlink entity for line card
mlxsw: core_linecards: Introduce per line card auxiliary device
mlxsw: core_linecard_dev: Set nested devlink relationship for a line
card
mlxsw: core_linecards: Expose HW revision and INI version
mlxsw: reg: Extend MDDQ by device_info
mlxsw: core_linecards: Probe provisioned line cards for devices and
expose FW version
mlxsw: reg: Add Management DownStream Device Tunneling Register
mlxsw: core_linecards: Expose device PSID over device info
mlxsw: core_linecards: Implement line card device flashing
selftests: mlxsw: Check line card info on provisioned line card
selftests: mlxsw: Check line card info on activated line card

 Documentation/networking/devlink/mlxsw.rst    |  24 ++
 drivers/net/ethernet/mellanox/mlxsw/Kconfig   |   1 +
 drivers/net/ethernet/mellanox/mlxsw/Makefile  |   2 +-
 drivers/net/ethernet/mellanox/mlxsw/core.c    |  44 +-
 drivers/net/ethernet/mellanox/mlxsw/core.h    |  35 ++
 .../mellanox/mlxsw/core_linecard_dev.c        | 180 ++++++++
 .../ethernet/mellanox/mlxsw/core_linecards.c  | 403 ++++++++++++++++++
 drivers/net/ethernet/mellanox/mlxsw/reg.h     | 173 +++++++-
 include/net/devlink.h                         |   2 +
 include/uapi/linux/devlink.h                  |   2 +
 net/core/devlink.c                            |  42 ++
 .../drivers/net/mlxsw/devlink_linecard.sh     |  54 +++
 12 files changed, 948 insertions(+), 14 deletions(-)
 create mode 100644 drivers/net/ethernet/mellanox/mlxsw/core_linecard_dev.c
Signature · 1 line
2.35.3
author maintainer
#1 lore raw

PATCH 1/11 devlink: introduce nested devlink entity for line card

From: Jiri Pirko <[email protected]>

For the purpose of exposing device info and allow flash updated which is going to be implemented in follow-up patches, introduce a possibility for a line card to expose relation to nested devlink entity. The nested devlink entity represents the line card.

Example:

$ devlink lc show pci/0000:01:00.0 lc 1
pci/0000:01:00.0:
  lc 1 state active type 16x100G nested_devlink auxiliary/mlxsw_core.lc.0
    supported_types:
       16x100G
$ devlink dev show auxiliary/mlxsw_core.lc.0
auxiliary/mlxsw_core.lc.0
---
 include/net/devlink.h        |  2 ++
 include/uapi/linux/devlink.h |  2 ++
 net/core/devlink.c           | 42 ++++++++++++++++++++++++++++++++++++
 3 files changed, 46 insertions(+)
include/net/devlink.h +2 -0
index 2a2a2a0c93f7..83e62943e1d4 100644
@@ -1584,6 +1584,8 @@ void devlink_linecard_provision_clear(struct devlink_linecard *linecard);
 void devlink_linecard_provision_fail(struct devlink_linecard *linecard);
 void devlink_linecard_activate(struct devlink_linecard *linecard);
 void devlink_linecard_deactivate(struct devlink_linecard *linecard);
+void devlink_linecard_nested_dl_set(struct devlink_linecard *linecard,
+				    struct devlink *nested_devlink);
 int devlink_sb_register(struct devlink *devlink, unsigned int sb_index,
 			u32 size, u16 ingress_pools_count,
 			u16 egress_pools_count, u16 ingress_tc_count,
include/uapi/linux/devlink.h +2 -0
index b3d40a5d72ff..541321695f52 100644
@@ -576,6 +576,8 @@ enum devlink_attr {
 	DEVLINK_ATTR_LINECARD_TYPE,		/* string */
 	DEVLINK_ATTR_LINECARD_SUPPORTED_TYPES,	/* nested */
 
+	DEVLINK_ATTR_NESTED_DEVLINK,		/* nested */
+
 	/* add new attributes above here, update the policy in devlink.c */
 
 	__DEVLINK_ATTR_MAX,
net/core/devlink.c +42 -0

78 lines not on this page · raw message

Signature · 1 line
2.35.3
maintainer
#2 lore raw

Re: [patch net-next 01/11] devlink: introduce nested devlink entity for line card

Quoted 3 lines · Jiri Pirko

From: Jiri Pirko <[email protected]>

For the purpose of exposing device info and allow flash updated which is

s/updated/update/

Quoted 112 lines

going to be implemented in follow-up patches, introduce a possibility for a line card to expose relation to nested devlink entity. The nested devlink entity represents the line card.

Example:

$ devlink lc show pci/0000:01:00.0 lc 1
pci/0000:01:00.0:
  lc 1 state active type 16x100G nested_devlink auxiliary/mlxsw_core.lc.0
    supported_types:
       16x100G
$ devlink dev show auxiliary/mlxsw_core.lc.0
auxiliary/mlxsw_core.lc.0
---
 include/net/devlink.h        |  2 ++
 include/uapi/linux/devlink.h |  2 ++
 net/core/devlink.c           | 42 ++++++++++++++++++++++++++++++++++++
 3 files changed, 46 insertions(+)
include/net/devlink.h +2 -0
index 2a2a2a0c93f7..83e62943e1d4 100644
@@ -1584,6 +1584,8 @@ void devlink_linecard_provision_clear(struct devlink_linecard *linecard);
 void devlink_linecard_provision_fail(struct devlink_linecard *linecard);
 void devlink_linecard_activate(struct devlink_linecard *linecard);
 void devlink_linecard_deactivate(struct devlink_linecard *linecard);
+void devlink_linecard_nested_dl_set(struct devlink_linecard *linecard,
+				    struct devlink *nested_devlink);
 int devlink_sb_register(struct devlink *devlink, unsigned int sb_index,
 			u32 size, u16 ingress_pools_count,
 			u16 egress_pools_count, u16 ingress_tc_count,
include/uapi/linux/devlink.h +2 -0
index b3d40a5d72ff..541321695f52 100644
@@ -576,6 +576,8 @@ enum devlink_attr {
 	DEVLINK_ATTR_LINECARD_TYPE,		/* string */
 	DEVLINK_ATTR_LINECARD_SUPPORTED_TYPES,	/* nested */
 
+	DEVLINK_ATTR_NESTED_DEVLINK,		/* nested */
+
 	/* add new attributes above here, update the policy in devlink.c */
 
 	__DEVLINK_ATTR_MAX,
net/core/devlink.c +25 -0
index db61f3a341cb..a5953cfe1baa 100644
@@ -87,6 +87,7 @@ struct devlink_linecard {
 	const char *type;
 	struct devlink_linecard_type *types;
 	unsigned int types_count;
+	struct devlink *nested_devlink;
 };
 
 /**
@@ -796,6 +797,24 @@ static int devlink_nl_put_handle(struct sk_buff *msg, struct devlink *devlink)
 	return 0;
 }
 
+static int devlink_nl_put_nested_handle(struct sk_buff *msg, struct devlink *devlink)
+{
+	struct nlattr *nested_attr;
+
+	nested_attr = nla_nest_start(msg, DEVLINK_ATTR_NESTED_DEVLINK);
+	if (!nested_attr)
+		return -EMSGSIZE;
+	if (devlink_nl_put_handle(msg, devlink))
+		goto nla_put_failure;
+
+	nla_nest_end(msg, nested_attr);
+	return 0;
+
+nla_put_failure:
+	nla_nest_cancel(msg, nested_attr);
+	return -EMSGSIZE;
+}
+
 struct devlink_reload_combination {
 	enum devlink_reload_action action;
 	enum devlink_reload_limit limit;
@@ -2100,6 +2119,10 @@ static int devlink_nl_linecard_fill(struct sk_buff *msg,
 		nla_nest_end(msg, attr);
 	}
 
+	if (linecard->nested_devlink &&
+	    devlink_nl_put_nested_handle(msg, linecard->nested_devlink))
+		goto nla_put_failure;
+
 	genlmsg_end(msg, hdr);
 	return 0;
 
@@ -10335,6 +10358,7 @@ EXPORT_SYMBOL_GPL(devlink_linecard_provision_set);
 void devlink_linecard_provision_clear(struct devlink_linecard *linecard)
 {
 	mutex_lock(&linecard->state_lock);
+	WARN_ON(linecard->nested_devlink);
 	linecard->state = DEVLINK_LINECARD_STATE_UNPROVISIONED;
 	linecard->type = NULL;
 	devlink_linecard_notify(linecard, DEVLINK_CMD_LINECARD_NEW);
@@ -10353,6 +10377,7 @@ EXPORT_SYMBOL_GPL(devlink_linecard_provision_clear);
 void devlink_linecard_provision_fail(struct devlink_linecard *linecard)
 {
 	mutex_lock(&linecard->state_lock);
+	WARN_ON(linecard->nested_devlink);
 	linecard->state = DEVLINK_LINECARD_STATE_PROVISIONING_FAILED;
 	devlink_linecard_notify(linecard, DEVLINK_CMD_LINECARD_NEW);
 	mutex_unlock(&linecard->state_lock);

net/core/devlink.c void devlink_linecard_deactivate(struct devlink_linecard *linecard) · line 10429

Quoted 5 lines
net/core/devlink.c +2 -0
@@ -10400,6 +10425,23 @@ void devlink_linecard_deactivate(struct devlink_linecard *linecard)
 }
 EXPORT_SYMBOL_GPL(devlink_linecard_deactivate);
 
+/**
+ *	devlink_linecard_nested_dl_set - Attach/detach nested delink

s/delink/devlink/

net/core/devlink.c void devlink_linecard_deactivate(struct devlink_linecard *linecard) · line 10433

Quoted 4 lines
net/core/devlink.c +4 -0
+ *					 instance to linecard.
+ *
+ *	@linecard: devlink linecard
+ *      @nested_devlink: devlink instance to attach or NULL to detach

The alignment looks off

Quoted 17 lines
net/core/devlink.c +11 -0
+ */
+void devlink_linecard_nested_dl_set(struct devlink_linecard *linecard,
+				    struct devlink *nested_devlink)
+{
+	mutex_lock(&linecard->state_lock);
+	linecard->nested_devlink = nested_devlink;
+	devlink_linecard_notify(linecard, DEVLINK_CMD_LINECARD_NEW);
+	mutex_unlock(&linecard->state_lock);
+}
+EXPORT_SYMBOL_GPL(devlink_linecard_nested_dl_set);
+
 int devlink_sb_register(struct devlink *devlink, unsigned int sb_index,
 			u32 size, u16 ingress_pools_count,
 			u16 egress_pools_count, u16 ingress_tc_count,
Signature · 1 line
2.35.3
author maintainer
#3 lore raw

Re: [patch net-next 01/11] devlink: introduce nested devlink entity for line card

Quoted 127 lines · Wed, Jun 15, 2022 at 04:05:13PM CEST, [email protected] wrote:
Quoted 3 lines · Jiri Pirko

From: Jiri Pirko <[email protected]>

For the purpose of exposing device info and allow flash updated which is

s/updated/update/

Quoted 112 lines

going to be implemented in follow-up patches, introduce a possibility for a line card to expose relation to nested devlink entity. The nested devlink entity represents the line card.

Example:

$ devlink lc show pci/0000:01:00.0 lc 1
pci/0000:01:00.0:
  lc 1 state active type 16x100G nested_devlink auxiliary/mlxsw_core.lc.0
    supported_types:
       16x100G
$ devlink dev show auxiliary/mlxsw_core.lc.0
auxiliary/mlxsw_core.lc.0
---
 include/net/devlink.h        |  2 ++
 include/uapi/linux/devlink.h |  2 ++
 net/core/devlink.c           | 42 ++++++++++++++++++++++++++++++++++++
 3 files changed, 46 insertions(+)
include/net/devlink.h +2 -0
index 2a2a2a0c93f7..83e62943e1d4 100644
@@ -1584,6 +1584,8 @@ void devlink_linecard_provision_clear(struct devlink_linecard *linecard);
 void devlink_linecard_provision_fail(struct devlink_linecard *linecard);
 void devlink_linecard_activate(struct devlink_linecard *linecard);
 void devlink_linecard_deactivate(struct devlink_linecard *linecard);
+void devlink_linecard_nested_dl_set(struct devlink_linecard *linecard,
+				    struct devlink *nested_devlink);
 int devlink_sb_register(struct devlink *devlink, unsigned int sb_index,
 			u32 size, u16 ingress_pools_count,
 			u16 egress_pools_count, u16 ingress_tc_count,
include/uapi/linux/devlink.h +2 -0
index b3d40a5d72ff..541321695f52 100644
@@ -576,6 +576,8 @@ enum devlink_attr {
 	DEVLINK_ATTR_LINECARD_TYPE,		/* string */
 	DEVLINK_ATTR_LINECARD_SUPPORTED_TYPES,	/* nested */
 
+	DEVLINK_ATTR_NESTED_DEVLINK,		/* nested */
+
 	/* add new attributes above here, update the policy in devlink.c */
 
 	__DEVLINK_ATTR_MAX,
net/core/devlink.c +25 -0
index db61f3a341cb..a5953cfe1baa 100644
@@ -87,6 +87,7 @@ struct devlink_linecard {
 	const char *type;
 	struct devlink_linecard_type *types;
 	unsigned int types_count;
+	struct devlink *nested_devlink;
 };
 
 /**
@@ -796,6 +797,24 @@ static int devlink_nl_put_handle(struct sk_buff *msg, struct devlink *devlink)
 	return 0;
 }
 
+static int devlink_nl_put_nested_handle(struct sk_buff *msg, struct devlink *devlink)
+{
+	struct nlattr *nested_attr;
+
+	nested_attr = nla_nest_start(msg, DEVLINK_ATTR_NESTED_DEVLINK);
+	if (!nested_attr)
+		return -EMSGSIZE;
+	if (devlink_nl_put_handle(msg, devlink))
+		goto nla_put_failure;
+
+	nla_nest_end(msg, nested_attr);
+	return 0;
+
+nla_put_failure:
+	nla_nest_cancel(msg, nested_attr);
+	return -EMSGSIZE;
+}
+
 struct devlink_reload_combination {
 	enum devlink_reload_action action;
 	enum devlink_reload_limit limit;
@@ -2100,6 +2119,10 @@ static int devlink_nl_linecard_fill(struct sk_buff *msg,
 		nla_nest_end(msg, attr);
 	}
 
+	if (linecard->nested_devlink &&
+	    devlink_nl_put_nested_handle(msg, linecard->nested_devlink))
+		goto nla_put_failure;
+
 	genlmsg_end(msg, hdr);
 	return 0;
 
@@ -10335,6 +10358,7 @@ EXPORT_SYMBOL_GPL(devlink_linecard_provision_set);
 void devlink_linecard_provision_clear(struct devlink_linecard *linecard)
 {
 	mutex_lock(&linecard->state_lock);
+	WARN_ON(linecard->nested_devlink);
 	linecard->state = DEVLINK_LINECARD_STATE_UNPROVISIONED;
 	linecard->type = NULL;
 	devlink_linecard_notify(linecard, DEVLINK_CMD_LINECARD_NEW);
@@ -10353,6 +10377,7 @@ EXPORT_SYMBOL_GPL(devlink_linecard_provision_clear);
 void devlink_linecard_provision_fail(struct devlink_linecard *linecard)
 {
 	mutex_lock(&linecard->state_lock);
+	WARN_ON(linecard->nested_devlink);
 	linecard->state = DEVLINK_LINECARD_STATE_PROVISIONING_FAILED;
 	devlink_linecard_notify(linecard, DEVLINK_CMD_LINECARD_NEW);
 	mutex_unlock(&linecard->state_lock);

net/core/devlink.c void devlink_linecard_deactivate(struct devlink_linecard *linecard) · line 10429

Quoted 5 lines
net/core/devlink.c +2 -0
@@ -10400,6 +10425,23 @@ void devlink_linecard_deactivate(struct devlink_linecard *linecard)
 }
 EXPORT_SYMBOL_GPL(devlink_linecard_deactivate);
 
+/**
+ *	devlink_linecard_nested_dl_set - Attach/detach nested delink

s/delink/devlink/

Quoted 6 lines

net/core/devlink.c void devlink_linecard_deactivate(struct devlink_linecard *linecard) · line 10433

Quoted 4 lines
net/core/devlink.c +4 -0
+ *					 instance to linecard.
+ *
+ *	@linecard: devlink linecard
+ *      @nested_devlink: devlink instance to attach or NULL to detach

The alignment looks off

Will fix all in v2.

Quoted 18 lines
Quoted 18 lines
net/core/devlink.c +11 -0
+ */
+void devlink_linecard_nested_dl_set(struct devlink_linecard *linecard,
+				    struct devlink *nested_devlink)
+{
+	mutex_lock(&linecard->state_lock);
+	linecard->nested_devlink = nested_devlink;
+	devlink_linecard_notify(linecard, DEVLINK_CMD_LINECARD_NEW);
+	mutex_unlock(&linecard->state_lock);
+}
+EXPORT_SYMBOL_GPL(devlink_linecard_nested_dl_set);
+
 int devlink_sb_register(struct devlink *devlink, unsigned int sb_index,
 			u32 size, u16 ingress_pools_count,
 			u16 egress_pools_count, u16 ingress_tc_count,
Signature · 1 line
2.35.3
maintainer Acked-by on v1 1/11
#4 lore raw

Re: [patch net-next 01/11] devlink: introduce nested devlink entity for line card

In reply to Jiri Pirko
Quoted 17 lines · On Tue, 14 Jun 2022 14:33:16 +0200 Jiri Pirko wrote:

From: Jiri Pirko <[email protected]>

For the purpose of exposing device info and allow flash updated which is going to be implemented in follow-up patches, introduce a possibility for a line card to expose relation to nested devlink entity. The nested devlink entity represents the line card.

Example:

$ devlink lc show pci/0000:01:00.0 lc 1
pci/0000:01:00.0:
  lc 1 state active type 16x100G nested_devlink auxiliary/mlxsw_core.lc.0
    supported_types:
       16x100G
$ devlink dev show auxiliary/mlxsw_core.lc.0
auxiliary/mlxsw_core.lc.0
Quoted 1 line

Thanks!

author maintainer
#5 lore raw

PATCH 2/11 mlxsw: core_linecards: Introduce per line card auxiliary device

In reply to Jiri Pirko

From: Jiri Pirko <[email protected]>

In order to be eventually able to expose line card gearbox version and possibility to flash FW, model the line card as a separate device on auxiliary bus.

---
 drivers/net/ethernet/mellanox/mlxsw/Kconfig   |   1 +
 drivers/net/ethernet/mellanox/mlxsw/Makefile  |   2 +-
 drivers/net/ethernet/mellanox/mlxsw/core.c    |  13 +-
 drivers/net/ethernet/mellanox/mlxsw/core.h    |  10 ++
 .../mellanox/mlxsw/core_linecard_dev.c        | 152 ++++++++++++++++++
 .../ethernet/mellanox/mlxsw/core_linecards.c  |  10 ++
 6 files changed, 185 insertions(+), 3 deletions(-)
 create mode 100644 drivers/net/ethernet/mellanox/mlxsw/core_linecard_dev.c
drivers/net/ethernet/mellanox/mlxsw/Kconfig +1 -0
index 4683312861ac..a510bf2cff2f 100644
@@ -7,6 +7,7 @@ config MLXSW_CORE
 	tristate "Mellanox Technologies Switch ASICs support"
 	select NET_DEVLINK
 	select MLXFW
+	select AUXILIARY_BUS
 	help
 	  This driver supports Mellanox Technologies Switch ASICs family.
 
drivers/net/ethernet/mellanox/mlxsw/Makefile +1 -1
index 1a465fd5d8b3..c9a773d3158b 100644
@@ -2,7 +2,7 @@
 obj-$(CONFIG_MLXSW_CORE)	+= mlxsw_core.o
 mlxsw_core-objs			:= core.o core_acl_flex_keys.o \
 				   core_acl_flex_actions.o core_env.o \
-				   core_linecards.o
+				   core_linecards.o core_linecard_dev.o
 mlxsw_core-$(CONFIG_MLXSW_CORE_HWMON) += core_hwmon.o
 mlxsw_core-$(CONFIG_MLXSW_CORE_THERMAL) += core_thermal.o
 obj-$(CONFIG_MLXSW_PCI)		+= mlxsw_pci.o
drivers/net/ethernet/mellanox/mlxsw/core.c +11 -2
index fc52832241b3..8864533281bd 100644
@@ -3331,9 +3331,15 @@ static int __init mlxsw_core_module_init(void)
 {
 	int err;
 
+	err = mlxsw_linecard_driver_register();
+	if (err)
+		return err;
+
 	mlxsw_wq = alloc_workqueue(mlxsw_core_driver_name, 0, 0);
-	if (!mlxsw_wq)
-		return -ENOMEM;
+	if (!mlxsw_wq) {
+		err = -ENOMEM;
+		goto err_alloc_workqueue;
+	}
 	mlxsw_owq = alloc_ordered_workqueue("%s_ordered", 0,
 					    mlxsw_core_driver_name);
 	if (!mlxsw_owq) {
@@ -3344,6 +3350,8 @@ static int __init mlxsw_core_module_init(void)
 
 err_alloc_ordered_workqueue:
 	destroy_workqueue(mlxsw_wq);
+err_alloc_workqueue:
+	mlxsw_linecard_driver_unregister();
 	return err;
 }
 
@@ -3351,6 +3359,7 @@ static void __exit mlxsw_core_module_exit(void)
 {
 	destroy_workqueue(mlxsw_owq);
 	destroy_workqueue(mlxsw_wq);
+	mlxsw_linecard_driver_unregister();
 }
 
 module_init(mlxsw_core_module_init);
drivers/net/ethernet/mellanox/mlxsw/core.h +10 -0
index c2a891287047..cda20a4fcbdb 100644
@@ -12,6 +12,7 @@
 #include <linux/skbuff.h>
 #include <linux/workqueue.h>
 #include <linux/net_namespace.h>
+#include <linux/auxiliary_bus.h>
 #include <net/devlink.h>
 
 #include "trap.h"
@@ -567,6 +568,8 @@ enum mlxsw_linecard_status_event_type {
 	MLXSW_LINECARD_STATUS_EVENT_TYPE_UNPROVISION,
 };
 
+struct mlxsw_linecard_bdev;
+
 struct mlxsw_linecard {
 	u8 slot_index;
 	struct mlxsw_linecards *linecards;
@@ -581,6 +584,7 @@ struct mlxsw_linecard {
 	   active:1;
 	u16 hw_revision;
 	u16 ini_version;
+	struct mlxsw_linecard_bdev *bdev;
 };
 
 struct mlxsw_linecard_types_info;
@@ -620,4 +624,10 @@ void mlxsw_linecards_event_ops_unregister(struct mlxsw_core *mlxsw_core,
 					  struct mlxsw_linecards_event_ops *ops,
 					  void *priv);
 
+int mlxsw_linecard_bdev_add(struct mlxsw_linecard *linecard);
+void mlxsw_linecard_bdev_del(struct mlxsw_linecard *linecard);
+
+int mlxsw_linecard_driver_register(void);
+void mlxsw_linecard_driver_unregister(void);
+
 #endif
drivers/net/ethernet/mellanox/mlxsw/core_linecard_dev.c new file +152 -0

152 lines not on this page · raw message

drivers/net/ethernet/mellanox/mlxsw/core_linecards.c +10 -0
index 5c9869dcf674..ae51944cde0c 100644
@@ -232,6 +232,7 @@ mlxsw_linecard_provision_set(struct mlxsw_linecard *linecard, u8 card_type,
 {
 	struct mlxsw_linecards *linecards = linecard->linecards;
 	const char *type;
+	int err;
 
 	type = mlxsw_linecard_types_lookup(linecards, card_type);
 	mlxsw_linecard_status_event_done(linecard,
@@ -252,6 +253,14 @@ mlxsw_linecard_provision_set(struct mlxsw_linecard *linecard, u8 card_type,
 	linecard->provisioned = true;
 	linecard->hw_revision = hw_revision;
 	linecard->ini_version = ini_version;
+
+	err = mlxsw_linecard_bdev_add(linecard);
+	if (err) {
+		linecard->provisioned = false;
+		mlxsw_linecard_provision_fail(linecard);
+		return err;
+	}
+
 	devlink_linecard_provision_set(linecard->devlink_linecard, type);
 	return 0;
 }
@@ -260,6 +269,7 @@ static void mlxsw_linecard_provision_clear(struct mlxsw_linecard *linecard)
 {
 	mlxsw_linecard_status_event_done(linecard,
 					 MLXSW_LINECARD_STATUS_EVENT_TYPE_UNPROVISION);
+	mlxsw_linecard_bdev_del(linecard);
 	linecard->provisioned = false;
 	devlink_linecard_provision_clear(linecard->devlink_linecard);
 }
Signature · 1 line
2.35.3
maintainer
#6 lore raw

Re: [patch net-next 02/11] mlxsw: core_linecards: Introduce per line card auxiliary device

Quoted 37 lines
drivers/net/ethernet/mellanox/mlxsw/core_linecard_dev.c new file +37 -0
+int mlxsw_linecard_bdev_add(struct mlxsw_linecard *linecard)
+{
+	struct mlxsw_linecard_bdev *linecard_bdev;
+	int err;
+	int id;
+
+	id = mlxsw_linecard_bdev_id_alloc();
+	if (id < 0)
+		return id;
+
+	linecard_bdev = kzalloc(sizeof(*linecard_bdev), GFP_KERNEL);
+	if (!linecard_bdev) {
+		mlxsw_linecard_bdev_id_free(id);
+		return -ENOMEM;
+	}
+	linecard_bdev->adev.id = id;
+	linecard_bdev->adev.name = MLXSW_LINECARD_DEV_ID_NAME;
+	linecard_bdev->adev.dev.release = mlxsw_linecard_bdev_release;
+	linecard_bdev->adev.dev.parent = linecard->linecards->bus_info->dev;
+	linecard_bdev->linecard = linecard;
+
+	err = auxiliary_device_init(&linecard_bdev->adev);
+	if (err) {
+		mlxsw_linecard_bdev_id_free(id);
+		kfree(linecard_bdev);
+		return err;
+	}
+
+	err = auxiliary_device_add(&linecard_bdev->adev);
+	if (err) {
+		auxiliary_device_uninit(&linecard_bdev->adev);
+		return err;
+	}
+
+	linecard->bdev = linecard_bdev;
+	return 0;
+}

[...]

Quoted 19 lines
drivers/net/ethernet/mellanox/mlxsw/core_linecard_dev.c new file +19 -0
+static int mlxsw_linecard_bdev_probe(struct auxiliary_device *adev,
+				     const struct auxiliary_device_id *id)
+{
+	struct mlxsw_linecard_bdev *linecard_bdev =
+			container_of(adev, struct mlxsw_linecard_bdev, adev);
+	struct mlxsw_linecard_dev *linecard_dev;
+	struct devlink *devlink;
+
+	devlink = devlink_alloc(&mlxsw_linecard_dev_devlink_ops,
+				sizeof(*linecard_dev), &adev->dev);
+	if (!devlink)
+		return -ENOMEM;
+	linecard_dev = devlink_priv(devlink);
+	linecard_dev->linecard = linecard_bdev->linecard;
+	linecard_bdev->linecard_dev = linecard_dev;
+
+	devlink_register(devlink);
+	return 0;
+}

[...]

drivers/net/ethernet/mellanox/mlxsw/core_linecards.c mlxsw_linecard_provision_set(struct mlxsw_linecard *linecard, u8 card_type, · line 257

Quoted 6 lines
drivers/net/ethernet/mellanox/mlxsw/core_linecards.c +2 -0
@@ -252,6 +253,14 @@ mlxsw_linecard_provision_set(struct mlxsw_linecard *linecard, u8 card_type,
 	linecard->provisioned = true;
 	linecard->hw_revision = hw_revision;
 	linecard->ini_version = ini_version;
+
+	err = mlxsw_linecard_bdev_add(linecard);

If a line card is already provisioned and we are reloading the primary devlink instance, isn't this going to deadlock on the global (not per-instance) devlink mutex? It is held throughout the reload operation and also taken in devlink_register()

My understanding of the auxiliary bus model is that after adding a device to the bus via auxiliary_device_add(), the probe() function of the auxiliary driver will be called. In our case, this function acquires the global devlink mutex in devlink_register().

Quoted 9 lines
drivers/net/ethernet/mellanox/mlxsw/core_linecards.c +6 -0
+	if (err) {
+		linecard->provisioned = false;
+		mlxsw_linecard_provision_fail(linecard);
+		return err;
+	}
+
 	devlink_linecard_provision_set(linecard->devlink_linecard, type);
 	return 0;
 }
author maintainer
#7 lore raw

Re: [patch net-next 02/11] mlxsw: core_linecards: Introduce per line card auxiliary device

Quoted 74 lines · Wed, Jun 15, 2022 at 04:52:13PM CEST, [email protected] wrote:
Quoted 37 lines
drivers/net/ethernet/mellanox/mlxsw/core_linecard_dev.c new file +37 -0
+int mlxsw_linecard_bdev_add(struct mlxsw_linecard *linecard)
+{
+	struct mlxsw_linecard_bdev *linecard_bdev;
+	int err;
+	int id;
+
+	id = mlxsw_linecard_bdev_id_alloc();
+	if (id < 0)
+		return id;
+
+	linecard_bdev = kzalloc(sizeof(*linecard_bdev), GFP_KERNEL);
+	if (!linecard_bdev) {
+		mlxsw_linecard_bdev_id_free(id);
+		return -ENOMEM;
+	}
+	linecard_bdev->adev.id = id;
+	linecard_bdev->adev.name = MLXSW_LINECARD_DEV_ID_NAME;
+	linecard_bdev->adev.dev.release = mlxsw_linecard_bdev_release;
+	linecard_bdev->adev.dev.parent = linecard->linecards->bus_info->dev;
+	linecard_bdev->linecard = linecard;
+
+	err = auxiliary_device_init(&linecard_bdev->adev);
+	if (err) {
+		mlxsw_linecard_bdev_id_free(id);
+		kfree(linecard_bdev);
+		return err;
+	}
+
+	err = auxiliary_device_add(&linecard_bdev->adev);
+	if (err) {
+		auxiliary_device_uninit(&linecard_bdev->adev);
+		return err;
+	}
+
+	linecard->bdev = linecard_bdev;
+	return 0;
+}

[...]

Quoted 19 lines
drivers/net/ethernet/mellanox/mlxsw/core_linecard_dev.c new file +19 -0
+static int mlxsw_linecard_bdev_probe(struct auxiliary_device *adev,
+				     const struct auxiliary_device_id *id)
+{
+	struct mlxsw_linecard_bdev *linecard_bdev =
+			container_of(adev, struct mlxsw_linecard_bdev, adev);
+	struct mlxsw_linecard_dev *linecard_dev;
+	struct devlink *devlink;
+
+	devlink = devlink_alloc(&mlxsw_linecard_dev_devlink_ops,
+				sizeof(*linecard_dev), &adev->dev);
+	if (!devlink)
+		return -ENOMEM;
+	linecard_dev = devlink_priv(devlink);
+	linecard_dev->linecard = linecard_bdev->linecard;
+	linecard_bdev->linecard_dev = linecard_dev;
+
+	devlink_register(devlink);
+	return 0;
+}

[...]

drivers/net/ethernet/mellanox/mlxsw/core_linecards.c mlxsw_linecard_provision_set(struct mlxsw_linecard *linecard, u8 card_type, · line 257

Quoted 6 lines
drivers/net/ethernet/mellanox/mlxsw/core_linecards.c +2 -0
@@ -252,6 +253,14 @@ mlxsw_linecard_provision_set(struct mlxsw_linecard *linecard, u8 card_type,
 	linecard->provisioned = true;
 	linecard->hw_revision = hw_revision;
 	linecard->ini_version = ini_version;
+
+	err = mlxsw_linecard_bdev_add(linecard);

If a line card is already provisioned and we are reloading the primary devlink instance, isn't this going to deadlock on the global (not per-instance) devlink mutex? It is held throughout the reload operation and also taken in devlink_register()

Quoted 4 lines

My understanding of the auxiliary bus model is that after adding a device to the bus via auxiliary_device_add(), the probe() function of the auxiliary driver will be called. In our case, this function acquires the global devlink mutex in devlink_register().

No, the line card auxdev is supposed to be removed during linecard_fini(). This, I forgot to add, will do in v2.

Quoted 10 lines
Quoted 10 lines
drivers/net/ethernet/mellanox/mlxsw/core_linecards.c +6 -0
+	if (err) {
+		linecard->provisioned = false;
+		mlxsw_linecard_provision_fail(linecard);
+		return err;
+	}
+
 	devlink_linecard_provision_set(linecard->devlink_linecard, type);
 	return 0;
 }
maintainer
#8 lore raw

Re: [patch net-next 02/11] mlxsw: core_linecards: Introduce per line card auxiliary device

Quoted 75 lines · Jiri Pirko
Quoted 74 lines · Wed, Jun 15, 2022 at 04:52:13PM CEST, [email protected] wrote:
Quoted 37 lines
drivers/net/ethernet/mellanox/mlxsw/core_linecard_dev.c new file +37 -0
+int mlxsw_linecard_bdev_add(struct mlxsw_linecard *linecard)
+{
+	struct mlxsw_linecard_bdev *linecard_bdev;
+	int err;
+	int id;
+
+	id = mlxsw_linecard_bdev_id_alloc();
+	if (id < 0)
+		return id;
+
+	linecard_bdev = kzalloc(sizeof(*linecard_bdev), GFP_KERNEL);
+	if (!linecard_bdev) {
+		mlxsw_linecard_bdev_id_free(id);
+		return -ENOMEM;
+	}
+	linecard_bdev->adev.id = id;
+	linecard_bdev->adev.name = MLXSW_LINECARD_DEV_ID_NAME;
+	linecard_bdev->adev.dev.release = mlxsw_linecard_bdev_release;
+	linecard_bdev->adev.dev.parent = linecard->linecards->bus_info->dev;
+	linecard_bdev->linecard = linecard;
+
+	err = auxiliary_device_init(&linecard_bdev->adev);
+	if (err) {
+		mlxsw_linecard_bdev_id_free(id);
+		kfree(linecard_bdev);
+		return err;
+	}
+
+	err = auxiliary_device_add(&linecard_bdev->adev);
+	if (err) {
+		auxiliary_device_uninit(&linecard_bdev->adev);
+		return err;
+	}
+
+	linecard->bdev = linecard_bdev;
+	return 0;
+}

[...]

Quoted 19 lines
drivers/net/ethernet/mellanox/mlxsw/core_linecard_dev.c new file +19 -0
+static int mlxsw_linecard_bdev_probe(struct auxiliary_device *adev,
+				     const struct auxiliary_device_id *id)
+{
+	struct mlxsw_linecard_bdev *linecard_bdev =
+			container_of(adev, struct mlxsw_linecard_bdev, adev);
+	struct mlxsw_linecard_dev *linecard_dev;
+	struct devlink *devlink;
+
+	devlink = devlink_alloc(&mlxsw_linecard_dev_devlink_ops,
+				sizeof(*linecard_dev), &adev->dev);
+	if (!devlink)
+		return -ENOMEM;
+	linecard_dev = devlink_priv(devlink);
+	linecard_dev->linecard = linecard_bdev->linecard;
+	linecard_bdev->linecard_dev = linecard_dev;
+
+	devlink_register(devlink);
+	return 0;
+}

[...]

drivers/net/ethernet/mellanox/mlxsw/core_linecards.c mlxsw_linecard_provision_set(struct mlxsw_linecard *linecard, u8 card_type, · line 257

Quoted 6 lines
drivers/net/ethernet/mellanox/mlxsw/core_linecards.c +2 -0
@@ -252,6 +253,14 @@ mlxsw_linecard_provision_set(struct mlxsw_linecard *linecard, u8 card_type,
 	linecard->provisioned = true;
 	linecard->hw_revision = hw_revision;
 	linecard->ini_version = ini_version;
+
+	err = mlxsw_linecard_bdev_add(linecard);

If a line card is already provisioned and we are reloading the primary devlink instance, isn't this going to deadlock on the global (not per-instance) devlink mutex? It is held throughout the reload operation and also taken in devlink_register()

Quoted 7 lines
Quoted 4 lines

My understanding of the auxiliary bus model is that after adding a device to the bus via auxiliary_device_add(), the probe() function of the auxiliary driver will be called. In our case, this function acquires the global devlink mutex in devlink_register().

No, the line card auxdev is supposed to be removed during linecard_fini(). This, I forgot to add, will do in v2.

mlxsw_linecard_fini() is called as part of reload with the global devlink mutex held. The removal of the auxdev should prompt the unregistration of its devlink instance which also takes this mutex. If this doesn't deadlock, then I'm probably missing something.

Can you test reload with lockdep when line cards are already provisioned/active?

Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
drivers/net/ethernet/mellanox/mlxsw/core_linecards.c +6 -0
+	if (err) {
+		linecard->provisioned = false;
+		mlxsw_linecard_provision_fail(linecard);
+		return err;
+	}
+
 	devlink_linecard_provision_set(linecard->devlink_linecard, type);
 	return 0;
 }
author maintainer
#9 lore raw

Re: [patch net-next 02/11] mlxsw: core_linecards: Introduce per line card auxiliary device

Quoted 81 lines · Thu, Jun 16, 2022 at 09:11:56AM CEST, [email protected] wrote:
Quoted 80 lines · Jiri Pirko
Quoted 78 lines · Wed, Jun 15, 2022 at 04:52:13PM CEST, [email protected] wrote:
Quoted 37 lines
drivers/net/ethernet/mellanox/mlxsw/core_linecard_dev.c new file +37 -0
+int mlxsw_linecard_bdev_add(struct mlxsw_linecard *linecard)
+{
+	struct mlxsw_linecard_bdev *linecard_bdev;
+	int err;
+	int id;
+
+	id = mlxsw_linecard_bdev_id_alloc();
+	if (id < 0)
+		return id;
+
+	linecard_bdev = kzalloc(sizeof(*linecard_bdev), GFP_KERNEL);
+	if (!linecard_bdev) {
+		mlxsw_linecard_bdev_id_free(id);
+		return -ENOMEM;
+	}
+	linecard_bdev->adev.id = id;
+	linecard_bdev->adev.name = MLXSW_LINECARD_DEV_ID_NAME;
+	linecard_bdev->adev.dev.release = mlxsw_linecard_bdev_release;
+	linecard_bdev->adev.dev.parent = linecard->linecards->bus_info->dev;
+	linecard_bdev->linecard = linecard;
+
+	err = auxiliary_device_init(&linecard_bdev->adev);
+	if (err) {
+		mlxsw_linecard_bdev_id_free(id);
+		kfree(linecard_bdev);
+		return err;
+	}
+
+	err = auxiliary_device_add(&linecard_bdev->adev);
+	if (err) {
+		auxiliary_device_uninit(&linecard_bdev->adev);
+		return err;
+	}
+
+	linecard->bdev = linecard_bdev;
+	return 0;
+}

[...]

Quoted 19 lines
drivers/net/ethernet/mellanox/mlxsw/core_linecard_dev.c new file +19 -0
+static int mlxsw_linecard_bdev_probe(struct auxiliary_device *adev,
+				     const struct auxiliary_device_id *id)
+{
+	struct mlxsw_linecard_bdev *linecard_bdev =
+			container_of(adev, struct mlxsw_linecard_bdev, adev);
+	struct mlxsw_linecard_dev *linecard_dev;
+	struct devlink *devlink;
+
+	devlink = devlink_alloc(&mlxsw_linecard_dev_devlink_ops,
+				sizeof(*linecard_dev), &adev->dev);
+	if (!devlink)
+		return -ENOMEM;
+	linecard_dev = devlink_priv(devlink);
+	linecard_dev->linecard = linecard_bdev->linecard;
+	linecard_bdev->linecard_dev = linecard_dev;
+
+	devlink_register(devlink);
+	return 0;
+}

[...]

drivers/net/ethernet/mellanox/mlxsw/core_linecards.c mlxsw_linecard_provision_set(struct mlxsw_linecard *linecard, u8 card_type, · line 257

Quoted 6 lines
drivers/net/ethernet/mellanox/mlxsw/core_linecards.c +2 -0
@@ -252,6 +253,14 @@ mlxsw_linecard_provision_set(struct mlxsw_linecard *linecard, u8 card_type,
 	linecard->provisioned = true;
 	linecard->hw_revision = hw_revision;
 	linecard->ini_version = ini_version;
+
+	err = mlxsw_linecard_bdev_add(linecard);

If a line card is already provisioned and we are reloading the primary devlink instance, isn't this going to deadlock on the global (not per-instance) devlink mutex? It is held throughout the reload operation and also taken in devlink_register()

My understanding of the auxiliary bus model is that after adding a device to the bus via auxiliary_device_add(), the probe() function of the auxiliary driver will be called. In our case, this function acquires the global devlink mutex in devlink_register().

Quoted 7 lines
Quoted 2 lines

No, the line card auxdev is supposed to be removed during linecard_fini(). This, I forgot to add, will do in v2.

mlxsw_linecard_fini() is called as part of reload with the global devlink mutex held. The removal of the auxdev should prompt the unregistration of its devlink instance which also takes this mutex. If this doesn't deadlock, then I'm probably missing something.

You don't miss anything, it really does. Need to remove devlink_mutex first.

Quoted 16 lines

Can you test reload with lockdep when line cards are already provisioned/active?

Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
drivers/net/ethernet/mellanox/mlxsw/core_linecards.c +6 -0
+	if (err) {
+		linecard->provisioned = false;
+		mlxsw_linecard_provision_fail(linecard);
+		return err;
+	}
+
 	devlink_linecard_provision_set(linecard->devlink_linecard, type);
 	return 0;
 }
maintainer
#10 lore raw

Re: [patch net-next 02/11] mlxsw: core_linecards: Introduce per line card auxiliary device

Quoted 85 lines · Jiri Pirko
Quoted 84 lines · Thu, Jun 16, 2022 at 09:11:56AM CEST, [email protected] wrote:
Quoted 82 lines · Jiri Pirko
Quoted 74 lines · Wed, Jun 15, 2022 at 04:52:13PM CEST, [email protected] wrote:
Quoted 37 lines
drivers/net/ethernet/mellanox/mlxsw/core_linecard_dev.c new file +37 -0
+int mlxsw_linecard_bdev_add(struct mlxsw_linecard *linecard)
+{
+	struct mlxsw_linecard_bdev *linecard_bdev;
+	int err;
+	int id;
+
+	id = mlxsw_linecard_bdev_id_alloc();
+	if (id < 0)
+		return id;
+
+	linecard_bdev = kzalloc(sizeof(*linecard_bdev), GFP_KERNEL);
+	if (!linecard_bdev) {
+		mlxsw_linecard_bdev_id_free(id);
+		return -ENOMEM;
+	}
+	linecard_bdev->adev.id = id;
+	linecard_bdev->adev.name = MLXSW_LINECARD_DEV_ID_NAME;
+	linecard_bdev->adev.dev.release = mlxsw_linecard_bdev_release;
+	linecard_bdev->adev.dev.parent = linecard->linecards->bus_info->dev;
+	linecard_bdev->linecard = linecard;
+
+	err = auxiliary_device_init(&linecard_bdev->adev);
+	if (err) {
+		mlxsw_linecard_bdev_id_free(id);
+		kfree(linecard_bdev);
+		return err;
+	}
+
+	err = auxiliary_device_add(&linecard_bdev->adev);
+	if (err) {
+		auxiliary_device_uninit(&linecard_bdev->adev);
+		return err;
+	}
+
+	linecard->bdev = linecard_bdev;
+	return 0;
+}

[...]

Quoted 19 lines
drivers/net/ethernet/mellanox/mlxsw/core_linecard_dev.c new file +19 -0
+static int mlxsw_linecard_bdev_probe(struct auxiliary_device *adev,
+				     const struct auxiliary_device_id *id)
+{
+	struct mlxsw_linecard_bdev *linecard_bdev =
+			container_of(adev, struct mlxsw_linecard_bdev, adev);
+	struct mlxsw_linecard_dev *linecard_dev;
+	struct devlink *devlink;
+
+	devlink = devlink_alloc(&mlxsw_linecard_dev_devlink_ops,
+				sizeof(*linecard_dev), &adev->dev);
+	if (!devlink)
+		return -ENOMEM;
+	linecard_dev = devlink_priv(devlink);
+	linecard_dev->linecard = linecard_bdev->linecard;
+	linecard_bdev->linecard_dev = linecard_dev;
+
+	devlink_register(devlink);
+	return 0;
+}

[...]

drivers/net/ethernet/mellanox/mlxsw/core_linecards.c mlxsw_linecard_provision_set(struct mlxsw_linecard *linecard, u8 card_type, · line 257

Quoted 6 lines
drivers/net/ethernet/mellanox/mlxsw/core_linecards.c +2 -0
@@ -252,6 +253,14 @@ mlxsw_linecard_provision_set(struct mlxsw_linecard *linecard, u8 card_type,
 	linecard->provisioned = true;
 	linecard->hw_revision = hw_revision;
 	linecard->ini_version = ini_version;
+
+	err = mlxsw_linecard_bdev_add(linecard);

If a line card is already provisioned and we are reloading the primary devlink instance, isn't this going to deadlock on the global (not per-instance) devlink mutex? It is held throughout the reload operation and also taken in devlink_register()

Quoted 4 lines

My understanding of the auxiliary bus model is that after adding a device to the bus via auxiliary_device_add(), the probe() function of the auxiliary driver will be called. In our case, this function acquires the global devlink mutex in devlink_register().

No, the line card auxdev is supposed to be removed during linecard_fini(). This, I forgot to add, will do in v2.

Quoted 7 lines
Quoted 4 lines

mlxsw_linecard_fini() is called as part of reload with the global devlink mutex held. The removal of the auxdev should prompt the unregistration of its devlink instance which also takes this mutex. If this doesn't deadlock, then I'm probably missing something.

You don't miss anything, it really does. Need to remove devlink_mutex first.

Can you please send it separately? Will probably need thorough review and testing...

The comment above devlink_mutex is: "An overall lock guarding every operation coming from userspace. It also guards devlink devices list and it is taken when driver registers/unregisters it.", but devlink does not have "parallel_ops" enabled, so maybe it's enough to only use this lock to protect the devlink devices list?

Quoted 18 lines
Quoted 18 lines

Can you test reload with lockdep when line cards are already provisioned/active?

Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
drivers/net/ethernet/mellanox/mlxsw/core_linecards.c +6 -0
+	if (err) {
+		linecard->provisioned = false;
+		mlxsw_linecard_provision_fail(linecard);
+		return err;
+	}
+
 	devlink_linecard_provision_set(linecard->devlink_linecard, type);
 	return 0;
 }
author maintainer
#11 lore raw

Re: [patch net-next 02/11] mlxsw: core_linecards: Introduce per line card auxiliary device

Quoted 91 lines · Thu, Jun 16, 2022 at 07:41:52PM CEST, [email protected] wrote:
Quoted 90 lines · Jiri Pirko
Quoted 88 lines · Thu, Jun 16, 2022 at 09:11:56AM CEST, [email protected] wrote:
Quoted 75 lines · Jiri Pirko
Quoted 74 lines · Wed, Jun 15, 2022 at 04:52:13PM CEST, [email protected] wrote:
Quoted 37 lines
drivers/net/ethernet/mellanox/mlxsw/core_linecard_dev.c new file +37 -0
+int mlxsw_linecard_bdev_add(struct mlxsw_linecard *linecard)
+{
+	struct mlxsw_linecard_bdev *linecard_bdev;
+	int err;
+	int id;
+
+	id = mlxsw_linecard_bdev_id_alloc();
+	if (id < 0)
+		return id;
+
+	linecard_bdev = kzalloc(sizeof(*linecard_bdev), GFP_KERNEL);
+	if (!linecard_bdev) {
+		mlxsw_linecard_bdev_id_free(id);
+		return -ENOMEM;
+	}
+	linecard_bdev->adev.id = id;
+	linecard_bdev->adev.name = MLXSW_LINECARD_DEV_ID_NAME;
+	linecard_bdev->adev.dev.release = mlxsw_linecard_bdev_release;
+	linecard_bdev->adev.dev.parent = linecard->linecards->bus_info->dev;
+	linecard_bdev->linecard = linecard;
+
+	err = auxiliary_device_init(&linecard_bdev->adev);
+	if (err) {
+		mlxsw_linecard_bdev_id_free(id);
+		kfree(linecard_bdev);
+		return err;
+	}
+
+	err = auxiliary_device_add(&linecard_bdev->adev);
+	if (err) {
+		auxiliary_device_uninit(&linecard_bdev->adev);
+		return err;
+	}
+
+	linecard->bdev = linecard_bdev;
+	return 0;
+}

[...]

Quoted 19 lines
drivers/net/ethernet/mellanox/mlxsw/core_linecard_dev.c new file +19 -0
+static int mlxsw_linecard_bdev_probe(struct auxiliary_device *adev,
+				     const struct auxiliary_device_id *id)
+{
+	struct mlxsw_linecard_bdev *linecard_bdev =
+			container_of(adev, struct mlxsw_linecard_bdev, adev);
+	struct mlxsw_linecard_dev *linecard_dev;
+	struct devlink *devlink;
+
+	devlink = devlink_alloc(&mlxsw_linecard_dev_devlink_ops,
+				sizeof(*linecard_dev), &adev->dev);
+	if (!devlink)
+		return -ENOMEM;
+	linecard_dev = devlink_priv(devlink);
+	linecard_dev->linecard = linecard_bdev->linecard;
+	linecard_bdev->linecard_dev = linecard_dev;
+
+	devlink_register(devlink);
+	return 0;
+}

[...]

drivers/net/ethernet/mellanox/mlxsw/core_linecards.c mlxsw_linecard_provision_set(struct mlxsw_linecard *linecard, u8 card_type, · line 257

Quoted 6 lines
drivers/net/ethernet/mellanox/mlxsw/core_linecards.c +2 -0
@@ -252,6 +253,14 @@ mlxsw_linecard_provision_set(struct mlxsw_linecard *linecard, u8 card_type,
 	linecard->provisioned = true;
 	linecard->hw_revision = hw_revision;
 	linecard->ini_version = ini_version;
+
+	err = mlxsw_linecard_bdev_add(linecard);

If a line card is already provisioned and we are reloading the primary devlink instance, isn't this going to deadlock on the global (not per-instance) devlink mutex? It is held throughout the reload operation and also taken in devlink_register()

Quoted 7 lines
Quoted 4 lines

My understanding of the auxiliary bus model is that after adding a device to the bus via auxiliary_device_add(), the probe() function of the auxiliary driver will be called. In our case, this function acquires the global devlink mutex in devlink_register().

No, the line card auxdev is supposed to be removed during linecard_fini(). This, I forgot to add, will do in v2.

mlxsw_linecard_fini() is called as part of reload with the global devlink mutex held. The removal of the auxdev should prompt the unregistration of its devlink instance which also takes this mutex. If this doesn't deadlock, then I'm probably missing something.

Quoted 5 lines
Quoted 2 lines

You don't miss anything, it really does. Need to remove devlink_mutex first.

Can you please send it separately? Will probably need thorough review and testing...

Sure.

Quoted 6 lines

The comment above devlink_mutex is: "An overall lock guarding every operation coming from userspace. It also guards devlink devices list and it is taken when driver registers/unregisters it.", but devlink does not have "parallel_ops" enabled, so maybe it's enough to only use this lock to protect the devlink devices list?

I'm preparing a patchset that will answer all your questions, lets move the discussion there.

Quoted 19 lines
Quoted 19 lines
Quoted 19 lines

Can you test reload with lockdep when line cards are already provisioned/active?

Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
drivers/net/ethernet/mellanox/mlxsw/core_linecards.c +6 -0
+	if (err) {
+		linecard->provisioned = false;
+		mlxsw_linecard_provision_fail(linecard);
+		return err;
+	}
+
 	devlink_linecard_provision_set(linecard->devlink_linecard, type);
 	return 0;
 }
author maintainer
#12 lore raw

PATCH 3/11 mlxsw: core_linecard_dev: Set nested devlink relationship for a line card

In reply to Jiri Pirko

From: Jiri Pirko <[email protected]>

For newly created line card device devlink instance, set the relationship with the parent line card object.

---
 drivers/net/ethernet/mellanox/mlxsw/core_linecard_dev.c | 4 ++++
 1 file changed, 4 insertions(+)
drivers/net/ethernet/mellanox/mlxsw/core_linecard_dev.c +4 -0
index af70d3f7a177..d12abd935ded 100644
@@ -102,6 +102,7 @@ static int mlxsw_linecard_bdev_probe(struct auxiliary_device *adev,
 {
 	struct mlxsw_linecard_bdev *linecard_bdev =
 			container_of(adev, struct mlxsw_linecard_bdev, adev);
+	struct mlxsw_linecard *linecard = linecard_bdev->linecard;
 	struct mlxsw_linecard_dev *linecard_dev;
 	struct devlink *devlink;
 
@@ -114,6 +115,7 @@ static int mlxsw_linecard_bdev_probe(struct auxiliary_device *adev,
 	linecard_bdev->linecard_dev = linecard_dev;
 
 	devlink_register(devlink);
+	devlink_linecard_nested_dl_set(linecard->devlink_linecard, devlink);
 	return 0;
 }
 
@@ -122,7 +124,9 @@ static void mlxsw_linecard_bdev_remove(struct auxiliary_device *adev)
 	struct mlxsw_linecard_bdev *linecard_bdev =
 			container_of(adev, struct mlxsw_linecard_bdev, adev);
 	struct devlink *devlink = priv_to_devlink(linecard_bdev->linecard_dev);
+	struct mlxsw_linecard *linecard = linecard_bdev->linecard;
 
+	devlink_linecard_nested_dl_set(linecard->devlink_linecard, NULL);
 	devlink_unregister(devlink);
 	devlink_free(devlink);
 }
Signature · 1 line
2.35.3
author maintainer
#13 lore raw

PATCH 4/11 mlxsw: core_linecards: Expose HW revision and INI version

In reply to Jiri Pirko

From: Jiri Pirko <[email protected]>

Implement info_get() to expose HW revision of a linecard and loaded INI version.

Example:

$ devlink dev info auxiliary/mlxsw_core.lc.0
auxiliary/mlxsw_core.lc.0:
  versions:
      fixed:
        hw.revision 0
      running:
        ini.version 4
---
 Documentation/networking/devlink/mlxsw.rst    | 18 ++++++++++++
 drivers/net/ethernet/mellanox/mlxsw/core.h    |  4 +++
 .../mellanox/mlxsw/core_linecard_dev.c        | 11 ++++++++
 .../ethernet/mellanox/mlxsw/core_linecards.c  | 28 +++++++++++++++++++
 4 files changed, 61 insertions(+)
Documentation/networking/devlink/mlxsw.rst +18 -0
index cf857cb4ba8f..aededcf68df4 100644
@@ -58,6 +58,24 @@ The ``mlxsw`` driver reports the following versions
      - running
      - Three digit firmware version
 
+Line card auxiliary device info versions
+========================================
+
+The ``mlxsw`` driver reports the following versions for line card auxiliary device
+
+.. list-table:: devlink info versions implemented
+   :widths: 5 5 90
+
+   * - Name
+     - Type
+     - Description
+   * - ``hw.revision``
+     - fixed
+     - The hardware revision for this line card
+   * - ``ini.version``
+     - running
+     - Version of line card INI loaded
+
 Driver-specific Traps
 =====================
 
drivers/net/ethernet/mellanox/mlxsw/core.h +4 -0
index cda20a4fcbdb..ea56bcc0a399 100644
@@ -605,6 +605,10 @@ mlxsw_linecard_get(struct mlxsw_linecards *linecards, u8 slot_index)
 	return &linecards->linecards[slot_index - 1];
 }
 
+int mlxsw_linecard_devlink_info_get(struct mlxsw_linecard *linecard,
+				    struct devlink_info_req *req,
+				    struct netlink_ext_ack *extack);
+
 int mlxsw_linecards_init(struct mlxsw_core *mlxsw_core,
 			 const struct mlxsw_bus_info *bus_info);
 void mlxsw_linecards_fini(struct mlxsw_core *mlxsw_core);
drivers/net/ethernet/mellanox/mlxsw/core_linecard_dev.c +11 -0
index d12abd935ded..30659f8be41c 100644
@@ -94,7 +94,18 @@ void mlxsw_linecard_bdev_del(struct mlxsw_linecard *linecard)
 	auxiliary_device_uninit(&linecard_bdev->adev);
 }
 
+static int mlxsw_linecard_dev_devlink_info_get(struct devlink *devlink,
+					       struct devlink_info_req *req,
+					       struct netlink_ext_ack *extack)
+{
+	struct mlxsw_linecard_dev *linecard_dev = devlink_priv(devlink);
+	struct mlxsw_linecard *linecard = linecard_dev->linecard;
+
+	return mlxsw_linecard_devlink_info_get(linecard, req, extack);
+}
+
 static const struct devlink_ops mlxsw_linecard_dev_devlink_ops = {
+	.info_get			= mlxsw_linecard_dev_devlink_info_get,
 };
 
 static int mlxsw_linecard_bdev_probe(struct auxiliary_device *adev,
drivers/net/ethernet/mellanox/mlxsw/core_linecards.c +28 -0
index ae51944cde0c..c11d22708bba 100644
@@ -226,6 +226,34 @@ void mlxsw_linecards_event_ops_unregister(struct mlxsw_core *mlxsw_core,
 }
 EXPORT_SYMBOL(mlxsw_linecards_event_ops_unregister);
 
+int mlxsw_linecard_devlink_info_get(struct mlxsw_linecard *linecard,
+				    struct devlink_info_req *req,
+				    struct netlink_ext_ack *extack)
+{
+	char buf[32];
+	int err;
+
+	mutex_lock(&linecard->lock);
+	if (WARN_ON(!linecard->provisioned)) {
+		err = 0;
+		goto unlock;
+	}
+
+	sprintf(buf, "%d", linecard->hw_revision);
+	err = devlink_info_version_fixed_put(req, "hw.revision", buf);
+	if (err)
+		goto unlock;
+
+	sprintf(buf, "%d", linecard->ini_version);
+	err = devlink_info_version_running_put(req, "ini.version", buf);
+	if (err)
+		goto unlock;
+
+unlock:
+	mutex_unlock(&linecard->lock);
+	return err;
+}
+
 static int
 mlxsw_linecard_provision_set(struct mlxsw_linecard *linecard, u8 card_type,
 			     u16 hw_revision, u16 ini_version)
Signature · 1 line
2.35.3
author maintainer
#14 lore raw

PATCH 5/11 mlxsw: reg: Extend MDDQ by device_info

In reply to Jiri Pirko

From: Jiri Pirko <[email protected]>

Extend existing MDDQ register by possibility to query information about devices residing on a line card.

---
 drivers/net/ethernet/mellanox/mlxsw/reg.h | 83 ++++++++++++++++++++++-
 1 file changed, 82 insertions(+), 1 deletion(-)
drivers/net/ethernet/mellanox/mlxsw/reg.h +82 -1
index 93af6c974ece..c8d526669522 100644
@@ -11643,7 +11643,11 @@ MLXSW_ITEM32(reg, mddq, sie, 0x00, 31, 1);
 
 enum mlxsw_reg_mddq_query_type {
 	MLXSW_REG_MDDQ_QUERY_TYPE_SLOT_INFO = 1,
-	MLXSW_REG_MDDQ_QUERY_TYPE_SLOT_NAME = 3,
+	MLXSW_REG_MDDQ_QUERY_TYPE_DEVICE_INFO, /* If there are no devices
+						* on the slot, data_valid
+						* will be '0'.
+						*/
+	MLXSW_REG_MDDQ_QUERY_TYPE_SLOT_NAME,
 };
 
 /* reg_mddq_query_type
@@ -11657,6 +11661,28 @@ MLXSW_ITEM32(reg, mddq, query_type, 0x00, 16, 8);
  */
 MLXSW_ITEM32(reg, mddq, slot_index, 0x00, 0, 4);
 
+/* reg_mddq_response_msg_seq
+ * Response message sequential number. For a specific request, the response
+ * message sequential number is the following one. In addition, the last
+ * message should be 0.
+ * Access: RO
+ */
+MLXSW_ITEM32(reg, mddq, response_msg_seq, 0x04, 16, 8);
+
+/* reg_mddq_request_msg_seq
+ * Request message sequential number.
+ * The first message number should be 0.
+ * Access: Index
+ */
+MLXSW_ITEM32(reg, mddq, request_msg_seq, 0x04, 0, 8);
+
+/* reg_mddq_data_valid
+ * If set, the data in the data field is valid and contain the information
+ * for the queried index.
+ * Access: RO
+ */
+MLXSW_ITEM32(reg, mddq, data_valid, 0x08, 31, 1);
+
 /* reg_mddq_slot_info_provisioned
  * If set, the INI file is applied and the card is provisioned.
  * Access: RO
@@ -11743,6 +11769,61 @@ mlxsw_reg_mddq_slot_info_unpack(const char *payload, u8 *p_slot_index,
 	*p_card_type = mlxsw_reg_mddq_slot_info_card_type_get(payload);
 }
 
+/* reg_mddq_device_info_flash_owner
+ * If set, the device is the flash owner. Otherwise, a shared flash
+ * is used by this device (another device is the flash owner).
+ * Access: RO
+ */
+MLXSW_ITEM32(reg, mddq, device_info_flash_owner, 0x10, 30, 1);
+
+/* reg_mddq_device_info_device_index
+ * Device index. The first device should number 0.
+ * Access: RO
+ */
+MLXSW_ITEM32(reg, mddq, device_info_device_index, 0x10, 0, 8);
+
+/* reg_mddq_device_info_fw_major
+ * Major FW version number.
+ * Access: RO
+ */
+MLXSW_ITEM32(reg, mddq, device_info_fw_major, 0x14, 16, 16);
+
+/* reg_mddq_device_info_fw_minor
+ * Minor FW version number.
+ * Access: RO
+ */
+MLXSW_ITEM32(reg, mddq, device_info_fw_minor, 0x18, 16, 16);
+
+/* reg_mddq_device_info_fw_sub_minor
+ * Sub-minor FW version number.
+ * Access: RO
+ */
+MLXSW_ITEM32(reg, mddq, device_info_fw_sub_minor, 0x18, 0, 16);
+
+static inline void
+mlxsw_reg_mddq_device_info_pack(char *payload, u8 slot_index,
+				u8 request_msg_seq)
+{
+	__mlxsw_reg_mddq_pack(payload, slot_index,
+			      MLXSW_REG_MDDQ_QUERY_TYPE_DEVICE_INFO);
+	mlxsw_reg_mddq_request_msg_seq_set(payload, request_msg_seq);
+}
+
+static inline void
+mlxsw_reg_mddq_device_info_unpack(const char *payload, u8 *p_response_msg_seq,
+				  bool *p_data_valid, bool *p_flash_owner,
+				  u8 *p_device_index, u16 *p_fw_major,
+				  u16 *p_fw_minor, u16 *p_fw_sub_minor)
+{
+	*p_response_msg_seq = mlxsw_reg_mddq_response_msg_seq_get(payload);
+	*p_data_valid = mlxsw_reg_mddq_data_valid_get(payload);
+	*p_flash_owner = mlxsw_reg_mddq_device_info_flash_owner_get(payload);
+	*p_device_index = mlxsw_reg_mddq_device_info_device_index_get(payload);
+	*p_fw_major = mlxsw_reg_mddq_device_info_fw_major_get(payload);
+	*p_fw_minor = mlxsw_reg_mddq_device_info_fw_minor_get(payload);
+	*p_fw_sub_minor = mlxsw_reg_mddq_device_info_fw_sub_minor_get(payload);
+}
+
 #define MLXSW_REG_MDDQ_SLOT_ASCII_NAME_LEN 20
 
 /* reg_mddq_slot_ascii_name
Signature · 1 line
2.35.3
author maintainer
#15 lore raw

PATCH 6/11 mlxsw: core_linecards: Probe provisioned line cards for devices and expose FW version

In reply to Jiri Pirko

From: Jiri Pirko <[email protected]>

In case the line card is provisioned, go over all possible existing devices (gearboxes) on it and expose FW version of the flashable one.

Example:

$ devlink dev info auxiliary/mlxsw_core.lc.0
auxiliary/mlxsw_core.lc.0:
  versions:
      fixed:
        hw.revision 0
      running:
        ini.version 4
        fw 19.2010.1312
---
 Documentation/networking/devlink/mlxsw.rst    |  3 +
 drivers/net/ethernet/mellanox/mlxsw/core.h    |  9 +++
 .../ethernet/mellanox/mlxsw/core_linecards.c  | 57 +++++++++++++++++++
 3 files changed, 69 insertions(+)
Documentation/networking/devlink/mlxsw.rst +3 -0
index aededcf68df4..65ceed98f94d 100644
@@ -75,6 +75,9 @@ The ``mlxsw`` driver reports the following versions for line card auxiliary devi
    * - ``ini.version``
      - running
      - Version of line card INI loaded
+   * - ``fw.version``
+     - running
+     - Three digit firmware version of line card device
 
 Driver-specific Traps
 =====================
drivers/net/ethernet/mellanox/mlxsw/core.h +9 -0
index ea56bcc0a399..edb0e1307d20 100644
@@ -570,6 +570,12 @@ enum mlxsw_linecard_status_event_type {
 
 struct mlxsw_linecard_bdev;
 
+struct mlxsw_linecard_device_info {
+	u16 fw_major;
+	u16 fw_minor;
+	u16 fw_sub_minor;
+};
+
 struct mlxsw_linecard {
 	u8 slot_index;
 	struct mlxsw_linecards *linecards;
@@ -585,6 +591,9 @@ struct mlxsw_linecard {
 	u16 hw_revision;
 	u16 ini_version;
 	struct mlxsw_linecard_bdev *bdev;
+	struct {
+		struct mlxsw_linecard_device_info info;
+	} device;
 };
 
 struct mlxsw_linecard_types_info;
drivers/net/ethernet/mellanox/mlxsw/core_linecards.c +57 -0

75 lines not on this page · raw message

Signature · 1 line
2.35.3
author maintainer
#16 lore raw

PATCH 7/11 mlxsw: reg: Add Management DownStream Device Tunneling Register

In reply to Jiri Pirko

From: Jiri Pirko <[email protected]>

The MDDT register allows to deliver query and request messages (PRM registers, commands) to a DownStream device.

---
 drivers/net/ethernet/mellanox/mlxsw/reg.h | 90 +++++++++++++++++++++++
 1 file changed, 90 insertions(+)
drivers/net/ethernet/mellanox/mlxsw/reg.h +90 -0
index c8d526669522..953790b9abd1 100644
@@ -11622,6 +11622,95 @@ mlxsw_reg_mbct_unpack(const char *payload, u8 *p_slot_index,
 		*p_fsm_state = mlxsw_reg_mbct_fsm_state_get(payload);
 }
 
+/* MDDT - Management DownStream Device Tunneling Register
+ * ------------------------------------------------------
+ * This register allows to deliver query and request messages (PRM registers,
+ * commands) to a DownStream device.
+ */
+#define MLXSW_REG_MDDT_ID 0x9160
+#define MLXSW_REG_MDDT_LEN 0x110
+
+MLXSW_REG_DEFINE(mddt, MLXSW_REG_MDDT_ID, MLXSW_REG_MDDT_LEN);
+
+/* reg_mddt_slot_index
+ * Slot index.
+ * Access: Index
+ */
+MLXSW_ITEM32(reg, mddt, slot_index, 0x00, 8, 4);
+
+/* reg_mddt_device_index
+ * Device index.
+ * Access: Index
+ */
+MLXSW_ITEM32(reg, mddt, device_index, 0x00, 0, 8);
+
+/* reg_mddt_read_size
+ * Read size in D-Words.
+ * Access: OP
+ */
+MLXSW_ITEM32(reg, mddt, read_size, 0x04, 24, 8);
+
+/* reg_mddt_write_size
+ * Write size in D-Words.
+ * Access: OP
+ */
+MLXSW_ITEM32(reg, mddt, write_size, 0x04, 16, 8);
+
+enum mlxsw_reg_mddt_status {
+	MLXSW_REG_MDDT_STATUS_OK,
+};
+
+/* reg_mddt_status
+ * Return code of the Downstream Device to the register that was sent.
+ * Access: RO
+ */
+MLXSW_ITEM32(reg, mddt, status, 0x0C, 24, 8);
+
+enum mlxsw_reg_mddt_method {
+	MLXSW_REG_MDDT_METHOD_QUERY,
+	MLXSW_REG_MDDT_METHOD_WRITE,
+};
+
+/* reg_mddt_method
+ * Access: OP
+ */
+MLXSW_ITEM32(reg, mddt, method, 0x0C, 22, 2);
+
+/* reg_mddt_register_id
+ * Access: Index
+ */
+MLXSW_ITEM32(reg, mddt, register_id, 0x0C, 0, 16);
+
+#define MLXSW_REG_MDDT_PAYLOAD_OFFSET 0x0C
+#define MLXSW_REG_MDDT_PRM_REGISTER_HEADER_LEN 4
+
+static inline char *mlxsw_reg_mddt_inner_payload(char *payload)
+{
+	return payload + MLXSW_REG_MDDT_PAYLOAD_OFFSET +
+	       MLXSW_REG_MDDT_PRM_REGISTER_HEADER_LEN;
+}
+
+static inline void mlxsw_reg_mddt_pack(char *payload, u8 slot_index,
+				       u8 device_index,
+				       enum mlxsw_reg_mddt_method method,
+				       const struct mlxsw_reg_info *reg,
+				       char **inner_payload)
+{
+	int len = reg->len + MLXSW_REG_MDDT_PRM_REGISTER_HEADER_LEN;
+
+	if (WARN_ON(len + MLXSW_REG_MDDT_PAYLOAD_OFFSET > MLXSW_REG_MDDT_LEN))
+		len = MLXSW_REG_MDDT_LEN - MLXSW_REG_MDDT_PAYLOAD_OFFSET;
+
+	MLXSW_REG_ZERO(mddt, payload);
+	mlxsw_reg_mddt_slot_index_set(payload, slot_index);
+	mlxsw_reg_mddt_device_index_set(payload, device_index);
+	mlxsw_reg_mddt_method_set(payload, method);
+	mlxsw_reg_mddt_register_id_set(payload, reg->id);
+	mlxsw_reg_mddt_read_size_set(payload, len / 4);
+	mlxsw_reg_mddt_write_size_set(payload, len / 4);
+	*inner_payload = mlxsw_reg_mddt_inner_payload(payload);
+}
+
 /* MDDQ - Management DownStream Device Query Register
  * --------------------------------------------------
  * This register allows to query the DownStream device properties. The desired
@@ -13208,6 +13297,7 @@ static const struct mlxsw_reg_info *mlxsw_reg_infos[] = {
 	MLXSW_REG(mfgd),
 	MLXSW_REG(mgpir),
 	MLXSW_REG(mbct),
+	MLXSW_REG(mddt),
 	MLXSW_REG(mddq),
 	MLXSW_REG(mddc),
 	MLXSW_REG(mfde),
Signature · 1 line
2.35.3
author maintainer
#17 lore raw

PATCH 8/11 mlxsw: core_linecards: Expose device PSID over device info

In reply to Jiri Pirko

From: Jiri Pirko <[email protected]>

Used tunneled MGIR to obtain PSID of line card device and extend device_info_get() op to fill up the info with that.

Example:

$ devlink dev info auxiliary/mlxsw_core.lc.0
auxiliary/mlxsw_core.lc.0:
  versions:
      fixed:
        hw.revision 0
        fw.psid MT_0000000749
      running:
        ini.version 4
        fw 19.2010.1312
---
 Documentation/networking/devlink/mlxsw.rst    |  3 ++
 drivers/net/ethernet/mellanox/mlxsw/core.h    |  1 +
 .../ethernet/mellanox/mlxsw/core_linecards.c  | 31 +++++++++++++++++++
 3 files changed, 35 insertions(+)
Documentation/networking/devlink/mlxsw.rst +3 -0
index 65ceed98f94d..433962225bd4 100644
@@ -75,6 +75,9 @@ The ``mlxsw`` driver reports the following versions for line card auxiliary devi
    * - ``ini.version``
      - running
      - Version of line card INI loaded
+   * - ``fw.psid``
+     - fixed
+     - Line card device PSID
    * - ``fw.version``
      - running
      - Three digit firmware version of line card device
drivers/net/ethernet/mellanox/mlxsw/core.h +1 -0
index edb0e1307d20..6b8bafc66090 100644
@@ -574,6 +574,7 @@ struct mlxsw_linecard_device_info {
 	u16 fw_major;
 	u16 fw_minor;
 	u16 fw_sub_minor;
+	char psid[MLXSW_REG_MGIR_FW_INFO_PSID_SIZE];
 };
 
 struct mlxsw_linecard {
drivers/net/ethernet/mellanox/mlxsw/core_linecards.c +31 -0
index 0f494ea755a9..fe3154c4f92f 100644
@@ -87,6 +87,27 @@ static const char *mlxsw_linecard_type_name(struct mlxsw_linecard *linecard)
 	return linecard->name;
 }
 
+static int mlxsw_linecard_device_psid_get(struct mlxsw_linecard *linecard,
+					  u8 device_index, char *psid)
+{
+	struct mlxsw_core *mlxsw_core = linecard->linecards->mlxsw_core;
+	char mddt_pl[MLXSW_REG_MDDT_LEN];
+	char *mgir_pl;
+	int err;
+
+	mlxsw_reg_mddt_pack(mddt_pl, linecard->slot_index, device_index,
+			    MLXSW_REG_MDDT_METHOD_QUERY,
+			    MLXSW_REG(mgir), &mgir_pl);
+
+	mlxsw_reg_mgir_pack(mgir_pl);
+	err = mlxsw_reg_query(mlxsw_core, MLXSW_REG(mddt), mddt_pl);
+	if (err)
+		return err;
+
+	mlxsw_reg_mgir_fw_info_psid_memcpy_from(mgir_pl, psid);
+	return 0;
+}
+
 static int mlxsw_linecard_device_info_update(struct mlxsw_linecard *linecard)
 {
 	struct mlxsw_core *mlxsw_core = linecard->linecards->mlxsw_core;
@@ -121,6 +142,12 @@ static int mlxsw_linecard_device_info_update(struct mlxsw_linecard *linecard)
 				      linecard->slot_index);
 			return 0;
 		}
+
+		err = mlxsw_linecard_device_psid_get(linecard, device_index,
+						     info.psid);
+		if (err)
+			return err;
+
 		linecard->device.info = info;
 		flashable_found = true;
 	} while (msg_seq);
@@ -293,6 +320,10 @@ int mlxsw_linecard_devlink_info_get(struct mlxsw_linecard *linecard,
 	if (linecard->ready) {
 		struct mlxsw_linecard_device_info *info = &linecard->device.info;
 
+		err = devlink_info_version_fixed_put(req,
+						     DEVLINK_INFO_VERSION_GENERIC_FW_PSID,
+						     info->psid);
+
 		sprintf(buf, "%u.%u.%u", info->fw_major, info->fw_minor,
 			info->fw_sub_minor);
 		err = devlink_info_version_running_put(req,
Signature · 1 line
2.35.3
author maintainer
#18 lore raw

PATCH 9/11 mlxsw: core_linecards: Implement line card device flashing

In reply to Jiri Pirko

From: Jiri Pirko <[email protected]>

Implement flash_update() devlink op for the line card devlink instance to allow user to update line card gearbox FW using MDDT register and mlxfw.

Example:

$ devlink dev flash auxiliary/mlxsw_core.lc.0 file mellanox/fw-AGB-rel-19_2010_1312-022-EVB.mfa2
---
 drivers/net/ethernet/mellanox/mlxsw/core.c    |  31 +-
 drivers/net/ethernet/mellanox/mlxsw/core.h    |  11 +
 .../mellanox/mlxsw/core_linecard_dev.c        |  13 +
 .../ethernet/mellanox/mlxsw/core_linecards.c  | 277 ++++++++++++++++++
 4 files changed, 322 insertions(+), 10 deletions(-)
drivers/net/ethernet/mellanox/mlxsw/core.c +21 -10
index 8864533281bd..cbde437ddec0 100644
@@ -951,6 +951,20 @@ static struct mlxsw_driver *mlxsw_core_driver_get(const char *kind)
 	return mlxsw_driver;
 }
 
+int mlxsw_core_fw_flash(struct mlxsw_core *mlxsw_core,
+			struct mlxfw_dev *mlxfw_dev,
+			const struct firmware *firmware,
+			struct netlink_ext_ack *extack)
+{
+	int err;
+
+	mlxsw_core->fw_flash_in_progress = true;
+	err = mlxfw_firmware_flash(mlxfw_dev, firmware, extack);
+	mlxsw_core->fw_flash_in_progress = false;
+
+	return err;
+}
+
 struct mlxsw_core_fw_info {
 	struct mlxfw_dev mlxfw_dev;
 	struct mlxsw_core *mlxsw_core;
@@ -1105,8 +1119,9 @@ static const struct mlxfw_dev_ops mlxsw_core_fw_mlxsw_dev_ops = {
 	.fsm_release		= mlxsw_core_fw_fsm_release,
 };
 
-static int mlxsw_core_fw_flash(struct mlxsw_core *mlxsw_core, const struct firmware *firmware,
-			       struct netlink_ext_ack *extack)
+static int mlxsw_core_dev_fw_flash(struct mlxsw_core *mlxsw_core,
+				   const struct firmware *firmware,
+				   struct netlink_ext_ack *extack)
 {
 	struct mlxsw_core_fw_info mlxsw_core_fw_info = {
 		.mlxfw_dev = {
@@ -1117,13 +1132,9 @@ static int mlxsw_core_fw_flash(struct mlxsw_core *mlxsw_core, const struct firmw
 		},
 		.mlxsw_core = mlxsw_core
 	};
-	int err;
 
-	mlxsw_core->fw_flash_in_progress = true;
-	err = mlxfw_firmware_flash(&mlxsw_core_fw_info.mlxfw_dev, firmware, extack);
-	mlxsw_core->fw_flash_in_progress = false;
-
-	return err;
+	return mlxsw_core_fw_flash(mlxsw_core, &mlxsw_core_fw_info.mlxfw_dev,
+				   firmware, extack);
 }
 
 static int mlxsw_core_fw_rev_validate(struct mlxsw_core *mlxsw_core,
@@ -1169,7 +1180,7 @@ static int mlxsw_core_fw_rev_validate(struct mlxsw_core *mlxsw_core,
 		return err;
 	}
 
-	err = mlxsw_core_fw_flash(mlxsw_core, firmware, NULL);
+	err = mlxsw_core_dev_fw_flash(mlxsw_core, firmware, NULL);
 	release_firmware(firmware);
 	if (err)
 		dev_err(mlxsw_bus_info->dev, "Could not upgrade firmware\n");
@@ -1187,7 +1198,7 @@ static int mlxsw_core_fw_flash_update(struct mlxsw_core *mlxsw_core,
 				      struct devlink_flash_update_params *params,
 				      struct netlink_ext_ack *extack)
 {
-	return mlxsw_core_fw_flash(mlxsw_core, params->fw, extack);
+	return mlxsw_core_dev_fw_flash(mlxsw_core, params->fw, extack);
 }
 
 static int mlxsw_core_devlink_param_fw_load_policy_validate(struct devlink *devlink, u32 id,
drivers/net/ethernet/mellanox/mlxsw/core.h +11 -0
index 6b8bafc66090..e518956efbbc 100644
@@ -19,6 +19,7 @@
 #include "reg.h"
 #include "cmd.h"
 #include "resources.h"
+#include "../mlxfw/mlxfw.h"
 
 enum mlxsw_core_resource_id {
 	MLXSW_CORE_RESOURCE_PORTS = 1,
@@ -48,6 +49,11 @@ mlxsw_core_fw_rev_minor_subminor_validate(const struct mlxsw_fw_rev *rev,
 int mlxsw_core_driver_register(struct mlxsw_driver *mlxsw_driver);
 void mlxsw_core_driver_unregister(struct mlxsw_driver *mlxsw_driver);
 
+int mlxsw_core_fw_flash(struct mlxsw_core *mlxsw_core,
+			struct mlxfw_dev *mlxfw_dev,
+			const struct firmware *firmware,
+			struct netlink_ext_ack *extack);
+
 int mlxsw_core_bus_device_register(const struct mlxsw_bus_info *mlxsw_bus_info,
 				   const struct mlxsw_bus *mlxsw_bus,
 				   void *bus_priv, bool reload,
@@ -594,6 +600,7 @@ struct mlxsw_linecard {
 	struct mlxsw_linecard_bdev *bdev;
 	struct {
 		struct mlxsw_linecard_device_info info;
+		u8 index;
 	} device;
 };
 
@@ -618,6 +625,10 @@ mlxsw_linecard_get(struct mlxsw_linecards *linecards, u8 slot_index)
 int mlxsw_linecard_devlink_info_get(struct mlxsw_linecard *linecard,
 				    struct devlink_info_req *req,
 				    struct netlink_ext_ack *extack);
+int mlxsw_linecard_flash_update(struct devlink *linecard_devlink,
+				struct mlxsw_linecard *linecard,
+				const struct firmware *firmware,
+				struct netlink_ext_ack *extack);
 
 int mlxsw_linecards_init(struct mlxsw_core *mlxsw_core,
 			 const struct mlxsw_bus_info *bus_info);
drivers/net/ethernet/mellanox/mlxsw/core_linecard_dev.c +13 -0
index 30659f8be41c..17496ba51081 100644
@@ -104,8 +104,21 @@ static int mlxsw_linecard_dev_devlink_info_get(struct devlink *devlink,
 	return mlxsw_linecard_devlink_info_get(linecard, req, extack);
 }
 
+static int
+mlxsw_linecard_dev_devlink_flash_update(struct devlink *devlink,
+					struct devlink_flash_update_params *params,
+					struct netlink_ext_ack *extack)
+{
+	struct mlxsw_linecard_dev *linecard_dev = devlink_priv(devlink);
+	struct mlxsw_linecard *linecard = linecard_dev->linecard;
+
+	return mlxsw_linecard_flash_update(devlink, linecard,
+					   params->fw, extack);
+}
+
 static const struct devlink_ops mlxsw_linecard_dev_devlink_ops = {
 	.info_get			= mlxsw_linecard_dev_devlink_info_get,
+	.flash_update			= mlxsw_linecard_dev_devlink_flash_update,
 };
 
 static int mlxsw_linecard_bdev_probe(struct auxiliary_device *adev,
drivers/net/ethernet/mellanox/mlxsw/core_linecards.c +277 -0

295 lines not on this page · raw message

Signature · 1 line
2.35.3
author maintainer
#19 lore raw

PATCH 10/11 selftests: mlxsw: Check line card info on provisioned line card

In reply to Jiri Pirko

From: Jiri Pirko <[email protected]>

Once line card is provisioned, check if HW revision and INI version are exposed on associated nested auxiliary device.

---
 .../drivers/net/mlxsw/devlink_linecard.sh     | 30 +++++++++++++++++++
 1 file changed, 30 insertions(+)
tools/testing/selftests/drivers/net/mlxsw/devlink_linecard.sh +30 -0
index 08a922d8b86a..ca4e9b08a105 100755
@@ -84,6 +84,13 @@ lc_wait_until_port_count_is()
 	busywait "$timeout" until_lc_port_count_is "$port_count" lc_port_count_get "$lc"
 }
 
+lc_nested_devlink_dev_get()
+{
+	local lc=$1
+
+	devlink lc show $DEVLINK_DEV lc $lc -j | jq -e -r ".[][][].nested_devlink"
+}
+
 PROV_UNPROV_TIMEOUT=8000 # ms
 POST_PROV_ACT_TIMEOUT=2000 # ms
 PROV_PORTS_INSTANTIATION_TIMEOUT=15000 # ms
@@ -191,12 +198,30 @@ ports_check()
 	check_err $? "Unexpected port count linecard $lc (got $port_count, expected $expected_port_count)"
 }
 
+lc_dev_info_provisioned_check()
+{
+	local lc=$1
+	local nested_devlink_dev=$2
+	local fixed_hw_revision
+	local running_ini_version
+
+	fixed_hw_revision=$(devlink dev info $nested_devlink_dev -j | \
+			    jq -e -r '.[][].versions.fixed."hw.revision"')
+	check_err $? "Failed to get linecard $lc fixed.hw.revision"
+	log_info "Linecard $lc fixed.hw.revision: \"$fixed_hw_revision\""
+	running_ini_version=$(devlink dev info $nested_devlink_dev -j | \
+			      jq -e -r '.[][].versions.running."ini.version"')
+	check_err $? "Failed to get linecard $lc running.ini.version"
+	log_info "Linecard $lc running.ini.version: \"$running_ini_version\""
+}
+
 provision_test()
 {
 	RET=0
 	local lc
 	local type
 	local state
+	local nested_devlink_dev
 
 	lc=$LC_SLOT
 	supported_types_check $lc
@@ -207,6 +232,11 @@ provision_test()
 	fi
 	provision_one $lc $LC_16X100G_TYPE
 	ports_check $lc $LC_16X100G_PORT_COUNT
+
+	nested_devlink_dev=$(lc_nested_devlink_dev_get $lc)
+	check_err $? "Failed to get nested devlink handle of linecard $lc"
+	lc_dev_info_provisioned_check $lc $nested_devlink_dev
+
 	log_test "Provision"
 }
 
Signature · 1 line
2.35.3
author maintainer
#20 lore raw

PATCH 11/11 selftests: mlxsw: Check line card info on activated line card

In reply to Jiri Pirko

From: Jiri Pirko <[email protected]>

Once line card is activated, check the FW version and PSID are exposed.

---
 .../drivers/net/mlxsw/devlink_linecard.sh     | 24 +++++++++++++++++++
 1 file changed, 24 insertions(+)
tools/testing/selftests/drivers/net/mlxsw/devlink_linecard.sh +24 -0
index ca4e9b08a105..224ca3695c89 100755
@@ -250,12 +250,32 @@ interface_check()
 	setup_wait
 }
 
+lc_dev_info_active_check()
+{
+	local lc=$1
+	local nested_devlink_dev=$2
+	local fixed_device_fw_psid
+	local running_device_fw
+
+	fixed_device_fw_psid=$(devlink dev info $nested_devlink_dev -j | \
+			       jq -e -r ".[][].versions.fixed" | \
+			       jq -e -r '."fw.psid"')
+	check_err $? "Failed to get linecard $lc fixed fw PSID"
+	log_info "Linecard $lc fixed.fw.psid: \"$fixed_device_fw_psid\""
+
+	running_device_fw=$(devlink dev info $nested_devlink_dev -j | \
+			    jq -e -r ".[][].versions.running.fw")
+	check_err $? "Failed to get linecard $lc running.fw.version"
+	log_info "Linecard $lc running.fw: \"$running_device_fw\""
+}
+
 activation_16x100G_test()
 {
 	RET=0
 	local lc
 	local type
 	local state
+	local nested_devlink_dev
 
 	lc=$LC_SLOT
 	type=$LC_16X100G_TYPE
@@ -268,6 +288,10 @@ activation_16x100G_test()
 
 	interface_check
 
+	nested_devlink_dev=$(lc_nested_devlink_dev_get $lc)
+	check_err $? "Failed to get nested devlink handle of linecard $lc"
+	lc_dev_info_active_check $lc $nested_devlink_dev
+
 	log_test "Activation 16x100G"
 }
 
Signature · 1 line
2.35.3
maintainer
#21 lore raw
In reply to Jiri Pirko
Quoted 9 lines · Jiri Pirko

From: Jiri Pirko <[email protected]>

This patchset implements two features:
1) "devlink dev info" is exposed for line card (patches 3-8)
2) "devlink dev flash" is implemented for line card gearbox flashing (patch 9)

For every line card, "a nested" auxiliary device is created which allows to bind the features mentioned above (patch 2).

The design choice to use an auxiliary device for each line card needs to be explained in the cover letter. From what I can tell, the motivation is to reuse the above devlink uAPI for line cards as opposed to using the "component" attribute or adding new uAPI. This is achieved by creating a devlink instance for each line card. The auxiliary device is needed because each devlink instance is expected to be associated with a device. Does this constitute proper use of the auxiliary bus?

Quoted 9 lines

The relationship between line card and its auxiliary dev devlink is carried over extra line card netlink attribute (patches 1 and 3).

Examples:

$ devlink lc show pci/0000:01:00.0 lc 1
pci/0000:01:00.0:
  lc 1 state active type 16x100G nested_devlink auxiliary/mlxsw_core.lc.0

Can we try to use the index of the line card as the identifier of the auxiliary device?

Quoted 5 lines
    supported_types:
       16x100G
$ devlink dev show auxiliary/mlxsw_core.lc.0
auxiliary/mlxsw_core.lc.0

I assume that the auxiliary device cannot outlive line card. Does that mean that as part of reload of the primary devlink instance the nested devlink instance is removed? If so, did you check the reload flow with lockdep to ensure there aren't any problems?

Quoted 11 lines
$ devlink dev info auxiliary/mlxsw_core.lc.0
auxiliary/mlxsw_core.lc.0:
  versions:
      fixed:
        hw.revision 0
        fw.psid MT_0000000749
      running:
        ini.version 4
        fw 19.2010.1312
Quoted 1 line
$ devlink dev flash auxiliary/mlxsw_core.lc.0 file mellanox/fw-AGB-rel-19_2010_1312-022-EVB.mfa2

How is this firmware activated? It is usually done after reload, but I don't see reload implementation for the line card devlink instance.

Quoted 34 lines

Jiri Pirko (11):
devlink: introduce nested devlink entity for line card
mlxsw: core_linecards: Introduce per line card auxiliary device
mlxsw: core_linecard_dev: Set nested devlink relationship for a line
card
mlxsw: core_linecards: Expose HW revision and INI version
mlxsw: reg: Extend MDDQ by device_info
mlxsw: core_linecards: Probe provisioned line cards for devices and
expose FW version
mlxsw: reg: Add Management DownStream Device Tunneling Register
mlxsw: core_linecards: Expose device PSID over device info
mlxsw: core_linecards: Implement line card device flashing
selftests: mlxsw: Check line card info on provisioned line card
selftests: mlxsw: Check line card info on activated line card

 Documentation/networking/devlink/mlxsw.rst    |  24 ++
 drivers/net/ethernet/mellanox/mlxsw/Kconfig   |   1 +
 drivers/net/ethernet/mellanox/mlxsw/Makefile  |   2 +-
 drivers/net/ethernet/mellanox/mlxsw/core.c    |  44 +-
 drivers/net/ethernet/mellanox/mlxsw/core.h    |  35 ++
 .../mellanox/mlxsw/core_linecard_dev.c        | 180 ++++++++
 .../ethernet/mellanox/mlxsw/core_linecards.c  | 403 ++++++++++++++++++
 drivers/net/ethernet/mellanox/mlxsw/reg.h     | 173 +++++++-
 include/net/devlink.h                         |   2 +
 include/uapi/linux/devlink.h                  |   2 +
 net/core/devlink.c                            |  42 ++
 .../drivers/net/mlxsw/devlink_linecard.sh     |  54 +++
 12 files changed, 948 insertions(+), 14 deletions(-)
 create mode 100644 drivers/net/ethernet/mellanox/mlxsw/core_linecard_dev.c
Signature · 1 line
2.35.3
author maintainer
#22 lore raw
Quoted 11 lines · Wed, Jun 15, 2022 at 11:13:35AM CEST, [email protected] wrote:
Quoted 9 lines · Jiri Pirko

From: Jiri Pirko <[email protected]>

This patchset implements two features:
1) "devlink dev info" is exposed for line card (patches 3-8)
2) "devlink dev flash" is implemented for line card gearbox flashing (patch 9)

For every line card, "a nested" auxiliary device is created which allows to bind the features mentioned above (patch 2).

Quoted 7 lines

The design choice to use an auxiliary device for each line card needs to be explained in the cover letter. From what I can tell, the motivation is to reuse the above devlink uAPI for line cards as opposed to using the "component" attribute or adding new uAPI. This is achieved by creating a devlink instance for each line card. The auxiliary device is needed because each devlink instance is expected to be associated with a device. Does this constitute proper use of the auxiliary bus?

Right, will describe this in cover letter.

Quoted 5 lines
Quoted 5 lines

The relationship between line card and its auxiliary dev devlink is carried over extra line card netlink attribute (patches 1 and 3).

Quoted 8 lines
Quoted 5 lines

Examples:

$ devlink lc show pci/0000:01:00.0 lc 1
pci/0000:01:00.0:
  lc 1 state active type 16x100G nested_devlink auxiliary/mlxsw_core.lc.0

Can we try to use the index of the line card as the identifier of the auxiliary device?

Not really. We would have a collision if there are 2 mlxsw instances.

Quoted 11 lines
Quoted 6 lines
    supported_types:
       16x100G
$ devlink dev show auxiliary/mlxsw_core.lc.0
auxiliary/mlxsw_core.lc.0

I assume that the auxiliary device cannot outlive line card. Does that mean that as part of reload of the primary devlink instance the nested devlink instance is removed? If so, did you check the reload flow with lockdep to ensure there aren't any problems?

As I wrote in the other email, line card auxdev should be removed during linecard_fini(), will add that to v2.

Quoted 12 lines
Quoted 12 lines

$ devlink dev info auxiliary/mlxsw_core.lc.0
auxiliary/mlxsw_core.lc.0:
  versions:
      fixed:
        hw.revision 0
        fw.psid MT_0000000749
      running:
        ini.version 4
        fw 19.2010.1312
Quoted 4 lines
Quoted 1 line
$ devlink dev flash auxiliary/mlxsw_core.lc.0 file mellanox/fw-AGB-rel-19_2010_1312-022-EVB.mfa2

How is this firmware activated? It is usually done after reload, but I don't see reload implementation for the line card devlink instance.

Currently, only devlink dev reload of the whole mlxsw instance or unprovision/provision of a line card.

Quoted 35 lines
Quoted 35 lines

Jiri Pirko (11):
devlink: introduce nested devlink entity for line card
mlxsw: core_linecards: Introduce per line card auxiliary device
mlxsw: core_linecard_dev: Set nested devlink relationship for a line
card
mlxsw: core_linecards: Expose HW revision and INI version
mlxsw: reg: Extend MDDQ by device_info
mlxsw: core_linecards: Probe provisioned line cards for devices and
expose FW version
mlxsw: reg: Add Management DownStream Device Tunneling Register
mlxsw: core_linecards: Expose device PSID over device info
mlxsw: core_linecards: Implement line card device flashing
selftests: mlxsw: Check line card info on provisioned line card
selftests: mlxsw: Check line card info on activated line card

 Documentation/networking/devlink/mlxsw.rst    |  24 ++
 drivers/net/ethernet/mellanox/mlxsw/Kconfig   |   1 +
 drivers/net/ethernet/mellanox/mlxsw/Makefile  |   2 +-
 drivers/net/ethernet/mellanox/mlxsw/core.c    |  44 +-
 drivers/net/ethernet/mellanox/mlxsw/core.h    |  35 ++
 .../mellanox/mlxsw/core_linecard_dev.c        | 180 ++++++++
 .../ethernet/mellanox/mlxsw/core_linecards.c  | 403 ++++++++++++++++++
 drivers/net/ethernet/mellanox/mlxsw/reg.h     | 173 +++++++-
 include/net/devlink.h                         |   2 +
 include/uapi/linux/devlink.h                  |   2 +
 net/core/devlink.c                            |  42 ++
 .../drivers/net/mlxsw/devlink_linecard.sh     |  54 +++
 12 files changed, 948 insertions(+), 14 deletions(-)
 create mode 100644 drivers/net/ethernet/mellanox/mlxsw/core_linecard_dev.c
Signature · 1 line
2.35.3
maintainer
#23 lore raw
Quoted 9 lines · Jiri Pirko
Quoted 5 lines · Wed, Jun 15, 2022 at 11:13:35AM CEST, [email protected] wrote:
Quoted 1 line · Jiri Pirko
$ devlink dev flash auxiliary/mlxsw_core.lc.0 file mellanox/fw-AGB-rel-19_2010_1312-022-EVB.mfa2

How is this firmware activated? It is usually done after reload, but I don't see reload implementation for the line card devlink instance.

Currently, only devlink dev reload of the whole mlxsw instance or unprovision/provision of a line card.

OK, please at least mention it in the commit message that adds flashing support.

What about implementing reload support as unprovision/provision?

author maintainer
#24 lore raw
Quoted 8 lines · Thu, Jun 16, 2022 at 09:03:52AM CEST, [email protected] wrote:
Quoted 7 lines · Jiri Pirko
Quoted 5 lines · Wed, Jun 15, 2022 at 11:13:35AM CEST, [email protected] wrote:
Quoted 1 line · Jiri Pirko
$ devlink dev flash auxiliary/mlxsw_core.lc.0 file mellanox/fw-AGB-rel-19_2010_1312-022-EVB.mfa2

How is this firmware activated? It is usually done after reload, but I don't see reload implementation for the line card devlink instance.

Quoted 7 lines
Quoted 2 lines

Currently, only devlink dev reload of the whole mlxsw instance or unprovision/provision of a line card.

OK, please at least mention it in the commit message that adds flashing support.

What about implementing reload support as unprovision/provision?

Yes, that can be done eventually. I was thinking about that as well.

maintainer
#25 lore raw
Quoted 12 lines · Jiri Pirko
Quoted 11 lines · Thu, Jun 16, 2022 at 09:03:52AM CEST, [email protected] wrote:
Quoted 9 lines · Jiri Pirko
Quoted 5 lines · Wed, Jun 15, 2022 at 11:13:35AM CEST, [email protected] wrote:
Quoted 1 line · Jiri Pirko
$ devlink dev flash auxiliary/mlxsw_core.lc.0 file mellanox/fw-AGB-rel-19_2010_1312-022-EVB.mfa2

How is this firmware activated? It is usually done after reload, but I don't see reload implementation for the line card devlink instance.

Currently, only devlink dev reload of the whole mlxsw instance or unprovision/provision of a line card.

Quoted 6 lines
Quoted 4 lines

OK, please at least mention it in the commit message that adds flashing support.

What about implementing reload support as unprovision/provision?

Yes, that can be done eventually. I was thinking about that as well.

This patch should come before the one that adds flashing. Then both the primary and nested devlink instances maintain the same semantics with regards to firmware flashing / activation.

author maintainer
#26 lore raw
Quoted 16 lines · Thu, Jun 16, 2022 at 04:47:50PM CEST, [email protected] wrote:
Quoted 15 lines · Jiri Pirko
Quoted 14 lines · Thu, Jun 16, 2022 at 09:03:52AM CEST, [email protected] wrote:
Quoted 9 lines · Jiri Pirko
Quoted 5 lines · Wed, Jun 15, 2022 at 11:13:35AM CEST, [email protected] wrote:
Quoted 1 line · Jiri Pirko
$ devlink dev flash auxiliary/mlxsw_core.lc.0 file mellanox/fw-AGB-rel-19_2010_1312-022-EVB.mfa2

How is this firmware activated? It is usually done after reload, but I don't see reload implementation for the line card devlink instance.

Currently, only devlink dev reload of the whole mlxsw instance or unprovision/provision of a line card.

OK, please at least mention it in the commit message that adds flashing support.

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

What about implementing reload support as unprovision/provision?

Yes, that can be done eventually. I was thinking about that as well.

This patch should come before the one that adds flashing. Then both the primary and nested devlink instances maintain the same semantics with regards to firmware flashing / activation.

Ok.

#27 lore raw
In reply to Jiri Pirko
Quoted 4 lines · Jiri Pirko
Quoted 3 lines · Wed, Jun 15, 2022 at 11:13:35AM CEST, [email protected] wrote:
Quoted 2 lines · Jiri Pirko

From: Jiri Pirko <[email protected]>

Quoted 8 lines
Quoted 8 lines
Quoted 7 lines

This patchset implements two features:
1) "devlink dev info" is exposed for line card (patches 3-8)
2) "devlink dev flash" is implemented for line card gearbox flashing (patch 9)

For every line card, "a nested" auxiliary device is created which allows to bind the features mentioned above (patch 2).

[...]>>

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

The relationship between line card and its auxiliary dev devlink is carried over extra line card netlink attribute (patches 1 and 3).

Examples:

Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
$ devlink lc show pci/0000:01:00.0 lc 1
pci/0000:01:00.0:
   lc 1 state active type 16x100G nested_devlink auxiliary/mlxsw_core.lc.0

Can we try to use the index of the line card as the identifier of the auxiliary device?

Not really. We would have a collision if there are 2 mlxsw instances.

Can you encode the base device's PCI info into the auxiliary device's id
to make it unique? Or maybe have each mlxsw instance have a unique ida value to encode in the linecard auxiliary device id?

sln

author maintainer
#28 lore raw
Quoted 26 lines · Sat, Jun 18, 2022 at 08:12:20AM CEST, [email protected] wrote:
Quoted 4 lines · Jiri Pirko
Quoted 3 lines · Wed, Jun 15, 2022 at 11:13:35AM CEST, [email protected] wrote:
Quoted 2 lines · Jiri Pirko

From: Jiri Pirko <[email protected]>

Quoted 8 lines
Quoted 8 lines
Quoted 7 lines

This patchset implements two features:
1) "devlink dev info" is exposed for line card (patches 3-8)
2) "devlink dev flash" is implemented for line card gearbox flashing (patch 9)

For every line card, "a nested" auxiliary device is created which allows to bind the features mentioned above (patch 2).

[...]>>

Quoted 10 lines
Quoted 10 lines
Quoted 9 lines

The relationship between line card and its auxiliary dev devlink is carried over extra line card netlink attribute (patches 1 and 3).

Examples:

$ devlink lc show pci/0000:01:00.0 lc 1
pci/0000:01:00.0:
   lc 1 state active type 16x100G nested_devlink auxiliary/mlxsw_core.lc.0
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

Can we try to use the index of the line card as the identifier of the auxiliary device?

Not really. We would have a collision if there are 2 mlxsw instances.

Can you encode the base device's PCI info into the auxiliary device's id

Would look odd to he PCI BDF in auxdev addsess, wouldn't it?

Quoted 2 lines

to make it unique? Or maybe have each mlxsw instance have a unique ida value to encode in the linecard auxiliary device id?

Well, which value would that bring? It would be dynamic random number. How the use would use that tho figure out the relation to the mlxsw instance?

Quoted 2 lines

sln

#29 lore raw
Quoted 30 lines · Jiri Pirko
Quoted 29 lines · Sat, Jun 18, 2022 at 08:12:20AM CEST, [email protected] wrote:
Quoted 4 lines · Jiri Pirko
Quoted 3 lines · Wed, Jun 15, 2022 at 11:13:35AM CEST, [email protected] wrote:
Quoted 2 lines · Jiri Pirko

From: Jiri Pirko <[email protected]>

Quoted 8 lines
Quoted 8 lines
Quoted 7 lines

This patchset implements two features:
1) "devlink dev info" is exposed for line card (patches 3-8)
2) "devlink dev flash" is implemented for line card gearbox flashing (patch 9)

For every line card, "a nested" auxiliary device is created which allows to bind the features mentioned above (patch 2).

[...]>>

Quoted 13 lines
Quoted 12 lines
Quoted 9 lines

The relationship between line card and its auxiliary dev devlink is carried over extra line card netlink attribute (patches 1 and 3).

Examples:

$ devlink lc show pci/0000:01:00.0 lc 1
pci/0000:01:00.0:
    lc 1 state active type 16x100G nested_devlink auxiliary/mlxsw_core.lc.0

Can we try to use the index of the line card as the identifier of the auxiliary device?

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

Not really. We would have a collision if there are 2 mlxsw instances.

Can you encode the base device's PCI info into the auxiliary device's id

Would look odd to he PCI BDF in auxdev addsess, wouldn't it?

Sure, it looks a little odd to see something like mycore.app.1281, but it does afford the auxiliary driver, and any other observer, a way to figure out which device it is representing. This also works nicely when trying to associate an auxiliary driver instance for a VF with the matching VF PCI driver instance.

sln

maintainer
#30 lore raw
Quoted 9 lines · On Mon, 27 Jun 2022 11:38:50 -0700 Shannon Nelson wrote:
Quoted 3 lines
Quoted 1 line

Can you encode the base device's PCI info into the auxiliary device's id

Would look odd to he PCI BDF in auxdev addsess, wouldn't it?

Sure, it looks a little odd to see something like mycore.app.1281, but it does afford the auxiliary driver, and any other observer, a way to figure out which device it is representing. This also works nicely when trying to associate an auxiliary driver instance for a VF with the matching VF PCI driver instance.

I'd personally not mind divorcing devlink from bus devices a little more. On one hand we have cases like this where there's naturally no bus device, on the other we have multi-link PCI devices which want to straddle NUMA nodes but otherwise are just a logical unit.

author maintainer
#31 lore raw
Quoted 5 lines · Mon, Jun 27, 2022 at 08:52:09PM CEST, [email protected] wrote:
Quoted 4 lines · On Mon, 27 Jun 2022 11:38:50 -0700 Shannon Nelson wrote:
Quoted 3 lines
Quoted 1 line

Can you encode the base device's PCI info into the auxiliary device's id

Would look odd to he PCI BDF in auxdev addsess, wouldn't it?

Quoted 8 lines
Quoted 5 lines

Sure, it looks a little odd to see something like mycore.app.1281, but it does afford the auxiliary driver, and any other observer, a way to figure out which device it is representing. This also works nicely when trying to associate an auxiliary driver instance for a VF with the matching VF PCI driver instance.

I'd personally not mind divorcing devlink from bus devices a little more. On one hand we have cases like this where there's naturally no

How exactly do you envision to do this? There is a good reason to have the handle based on bus/name, as it is constant and predictable.

Quoted 2 lines

bus device, on the other we have multi-link PCI devices which want to straddle NUMA nodes but otherwise are just a logical unit.