Skip to contentSkip to conversation

Devlink notification after recovery complete by bnxt_en driver

PATCH In review Vikas Gupta 2 participants 6 messages Jan 2, 2020 – Jan 8, 2020

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

cover Devlink notification after recovery complete by bnxt_en driver

Hi,
This patchset adds following feature in devlink
1) Recovery complete direct call API to be used by drivers when it successfully completes. It is required as recovery triggered by devlink may return with EINPROGRESS and eventually recovery completes in different context.
2) A notification when health status is updated by reporter.

Patchset also contains required changes in bnxt_en driver to mark recovery in progress when recovery is triggered from kernel devlink.

Vikas Gupta (3):
devlink: add support for reporter recovery completion
devlink: add devink notification when reporter update health state
bnxt_en: Call recovery done after reset is successfully done

 drivers/net/ethernet/broadcom/bnxt/bnxt.c         |  1 +
 drivers/net/ethernet/broadcom/bnxt/bnxt_devlink.c | 14 ++++-
 drivers/net/ethernet/broadcom/bnxt/bnxt_devlink.h |  1 +
 include/net/devlink.h                             |  2 +
 net/core/devlink.c                                | 62 +++++++++++++++++------
 5 files changed, 63 insertions(+), 17 deletions(-)
Signature · 1 line
2.7.4
#1 lore raw

PATCH 1/3 devlink: add support for reporter recovery completion

It is possible that a reporter recovery completion do not finish successfully when recovery is triggered via devlink_health_reporter_recover as recovery could be processed in different context. In such scenario an error is returned by driver when recover hook is invoked and successful recovery completion is intimated later.
Expose devlink recover done API to update recovery stats.

---
 include/net/devlink.h |  2 ++
 net/core/devlink.c    | 11 +++++++++--
 2 files changed, 11 insertions(+), 2 deletions(-)
include/net/devlink.h +2 -0
index 47f87b2..453f45c 100644
@@ -1000,6 +1000,8 @@ int devlink_health_report(struct devlink_health_reporter *reporter,
 void
 devlink_health_reporter_state_update(struct devlink_health_reporter *reporter,
 				     enum devlink_health_reporter_state state);
+void
+devlink_health_reporter_recovery_done(struct devlink_health_reporter *reporter);
 
 bool devlink_is_reload_failed(const struct devlink *devlink);
 
net/core/devlink.c +9 -2
index 4c63c9a..e686ae6 100644
@@ -4860,6 +4860,14 @@ devlink_health_reporter_state_update(struct devlink_health_reporter *reporter,
 }
 EXPORT_SYMBOL_GPL(devlink_health_reporter_state_update);
 
+void
+devlink_health_reporter_recovery_done(struct devlink_health_reporter *reporter)
+{
+	reporter->recovery_count++;
+	reporter->last_recovery_ts = jiffies;
+}
+EXPORT_SYMBOL_GPL(devlink_health_reporter_recovery_done);
+
 static int
 devlink_health_reporter_recover(struct devlink_health_reporter *reporter,
 				void *priv_ctx, struct netlink_ext_ack *extack)
@@ -4876,9 +4884,8 @@ devlink_health_reporter_recover(struct devlink_health_reporter *reporter,
 	if (err)
 		return err;
 
-	reporter->recovery_count++;
+	devlink_health_reporter_recovery_done(reporter);
 	reporter->health_state = DEVLINK_HEALTH_REPORTER_STATE_HEALTHY;
-	reporter->last_recovery_ts = jiffies;
 
 	return 0;
 }
Signature · 1 line
2.7.4
#2 lore raw

PATCH 2/3 devlink: add devink notification when reporter update health state

In reply to Vikas Gupta

add a devlink notification when reporter update the health state.

---
 net/core/devlink.c | 59 ++++++++++++++++++++++++++++++++++++++----------------
 1 file changed, 42 insertions(+), 17 deletions(-)
net/core/devlink.c +42 -17
index e686ae6..d30aa47 100644
@@ -4844,23 +4844,6 @@ devlink_health_reporter_destroy(struct devlink_health_reporter *reporter)
 EXPORT_SYMBOL_GPL(devlink_health_reporter_destroy);
 
 void
-devlink_health_reporter_state_update(struct devlink_health_reporter *reporter,
-				     enum devlink_health_reporter_state state)
-{
-	if (WARN_ON(state != DEVLINK_HEALTH_REPORTER_STATE_HEALTHY &&
-		    state != DEVLINK_HEALTH_REPORTER_STATE_ERROR))
-		return;
-
-	if (reporter->health_state == state)
-		return;
-
-	reporter->health_state = state;
-	trace_devlink_health_reporter_state_update(reporter->devlink,
-						   reporter->ops->name, state);
-}
-EXPORT_SYMBOL_GPL(devlink_health_reporter_state_update);
-
-void
 devlink_health_reporter_recovery_done(struct devlink_health_reporter *reporter)
 {
 	reporter->recovery_count++;
@@ -5097,6 +5080,48 @@ devlink_nl_health_reporter_fill(struct sk_buff *msg,
 	return -EMSGSIZE;
 }
 
+static void devlink_recover_notify(struct devlink_health_reporter *reporter,
+				   enum devlink_command cmd)
+{
+	struct sk_buff *msg;
+	int err;
+
+	WARN_ON(cmd != DEVLINK_CMD_HEALTH_REPORTER_RECOVER);
+
+	msg = nlmsg_new(NLMSG_DEFAULT_SIZE, GFP_KERNEL);
+	if (!msg)
+		return;
+
+	err = devlink_nl_health_reporter_fill(msg, reporter->devlink,
+					      reporter, cmd, 0, 0, 0);
+	if (err) {
+		nlmsg_free(msg);
+		return;
+	}
+
+	genlmsg_multicast_netns(&devlink_nl_family,
+				devlink_net(reporter->devlink),
+				msg, 0, DEVLINK_MCGRP_CONFIG, GFP_KERNEL);
+}
+
+void
+devlink_health_reporter_state_update(struct devlink_health_reporter *reporter,
+				     enum devlink_health_reporter_state state)
+{
+	if (WARN_ON(state != DEVLINK_HEALTH_REPORTER_STATE_HEALTHY &&
+		    state != DEVLINK_HEALTH_REPORTER_STATE_ERROR))
+		return;
+
+	if (reporter->health_state == state)
+		return;
+
+	reporter->health_state = state;
+	trace_devlink_health_reporter_state_update(reporter->devlink,
+						   reporter->ops->name, state);
+	devlink_recover_notify(reporter, DEVLINK_CMD_HEALTH_REPORTER_RECOVER);
+}
+EXPORT_SYMBOL_GPL(devlink_health_reporter_state_update);
+
 static int devlink_nl_cmd_health_reporter_get_doit(struct sk_buff *skb,
 						   struct genl_info *info)
 {
Signature · 1 line
2.7.4
#3 lore raw

PATCH 3/3 bnxt_en: Call recovery done after reset is successfully done

In reply to Vikas Gupta

Return EINPROGRESS to devlink health reporter recover as we are not yet done and call devlink_health_reporter_recovery_done once reset is successfully completed from workqueue context.

---
 drivers/net/ethernet/broadcom/bnxt/bnxt.c         |  1 +
 drivers/net/ethernet/broadcom/bnxt/bnxt_devlink.c | 14 ++++++++++++--
 drivers/net/ethernet/broadcom/bnxt/bnxt_devlink.h |  1 +
 3 files changed, 14 insertions(+), 2 deletions(-)
drivers/net/ethernet/broadcom/bnxt/bnxt.c +1 -0
index 7b0fe19..39d4309 100644
@@ -10822,6 +10822,7 @@ static void bnxt_fw_reset_task(struct work_struct *work)
 		smp_mb__before_atomic();
 		clear_bit(BNXT_STATE_IN_FW_RESET, &bp->state);
 		bnxt_ulp_start(bp, rc);
+		bnxt_dl_health_recovery_done(bp);
 		bnxt_dl_health_status_update(bp, true);
 		rtnl_unlock();
 		break;
drivers/net/ethernet/broadcom/bnxt/bnxt_devlink.c +12 -2
index 3eedd44..0c3d224 100644
@@ -89,7 +89,7 @@ static int bnxt_fw_reset_recover(struct devlink_health_reporter *reporter,
 		return -EOPNOTSUPP;
 
 	bnxt_fw_reset(bp);
-	return 0;
+	return -EINPROGRESS;
 }
 
 static const
@@ -116,7 +116,7 @@ static int bnxt_fw_fatal_recover(struct devlink_health_reporter *reporter,
 	else if (event == BNXT_FW_EXCEPTION_SP_EVENT)
 		bnxt_fw_exception(bp);
 
-	return 0;
+	return -EINPROGRESS;
 }
 
 static const
@@ -262,6 +262,16 @@ void bnxt_dl_health_status_update(struct bnxt *bp, bool healthy)
 	health->fatal = false;
 }
 
+void bnxt_dl_health_recovery_done(struct bnxt *bp)
+{
+	struct bnxt_fw_health *hlth = bp->fw_health;
+
+	if (hlth->fatal)
+		devlink_health_reporter_recovery_done(hlth->fw_fatal_reporter);
+	else
+		devlink_health_reporter_recovery_done(hlth->fw_reset_reporter);
+}
+
 static const struct devlink_ops bnxt_dl_ops = {
 #ifdef CONFIG_BNXT_SRIOV
 	.eswitch_mode_set = bnxt_dl_eswitch_mode_set,
drivers/net/ethernet/broadcom/bnxt/bnxt_devlink.h +1 -0
index 6db6c3d..08aaa44 100644
@@ -58,6 +58,7 @@ struct bnxt_dl_nvm_param {
 
 void bnxt_devlink_health_report(struct bnxt *bp, unsigned long event);
 void bnxt_dl_health_status_update(struct bnxt *bp, bool healthy);
+void bnxt_dl_health_recovery_done(struct bnxt *bp);
 void bnxt_dl_fw_reporters_create(struct bnxt *bp);
 void bnxt_dl_fw_reporters_destroy(struct bnxt *bp, bool all);
 int bnxt_dl_register(struct bnxt *bp);
Signature · 1 line
2.7.4
maintainer
#4 lore raw
In reply to Vikas Gupta

From: Vikas Gupta <[email protected]>
Date: Thu, 2 Jan 2020 21:18:08 +0530

Quoted 11 lines

Hi,
This patchset adds following feature in devlink
1) Recovery complete direct call API to be used by drivers when it successfully completes. It is required as recovery triggered by devlink may return with EINPROGRESS and eventually recovery completes in different context.
2) A notification when health status is updated by reporter.

Patchset also contains required changes in bnxt_en driver to mark recovery in progress when recovery is triggered from kernel devlink.

Jiri et al., please review this.

Thank you.

maintainer
#5 lore raw
In reply to Vikas Gupta

From: Vikas Gupta <[email protected]>
Date: Thu, 2 Jan 2020 21:18:08 +0530

Quoted 10 lines

This patchset adds following feature in devlink
1) Recovery complete direct call API to be used by drivers when it successfully completes. It is required as recovery triggered by devlink may return with EINPROGRESS and eventually recovery completes in different context.
2) A notification when health status is updated by reporter.

Patchset also contains required changes in bnxt_en driver to mark recovery in progress when recovery is triggered from kernel devlink.

Series applied to net-next, thanks.