Skip to contentSkip to conversation

devlink: expose instance locking and simplify port splitting

PATCH In review Jakub Kicinski 5 participants 37 messages Mar 10, 2022 – Mar 15, 2022

Subsystems (5)
lore
Share & tools
lore

Jakub Kicinski

author maintainer
#0 lore raw

cover devlink: expose instance locking and simplify port splitting

This series puts the devlink ports fully under the devlink instance lock's protection. As discussed in the past it implements my preferred solution of exposing the instance lock to the drivers. This way drivers which want to support port splitting can lock the devlink instance themselves on the probe path, and we can take that lock in the core on the split/unsplit paths.

nfp and mlxsw are converted, with slightly deeper changes done in nfp since I'm more familiar with that driver.

Now that the devlink port is protected we can pass a pointer to the drivers, instead of passing a port index and forcing the drivers to do their own lookups. Both nfp and mlxsw can container_of() to their own structures.

I'd appreciate some testing, I don't have access to this HW.

Jakub Kicinski (6):
devlink: expose instance locking and add locked port registering
eth: nfp: wrap locking assertions in helpers
eth: nfp: replace driver's "pf" lock with devlink instance lock
eth: mlxsw: switch to explicit locking for port registration
devlink: hold the instance lock in port_split / port_unsplit callbacks
devlink: pass devlink_port to port_split / port_unsplit callbacks

 drivers/net/ethernet/mellanox/mlxsw/core.c    |  36 ++---
 drivers/net/ethernet/mellanox/mlxsw/minimal.c |   6 +
 .../net/ethernet/mellanox/mlxsw/spectrum.c    |   7 +
 .../net/ethernet/netronome/nfp/flower/main.c  |   4 +-
 drivers/net/ethernet/netronome/nfp/nfp_app.c  |   2 +-
 drivers/net/ethernet/netronome/nfp/nfp_app.h  |  12 +-
 .../net/ethernet/netronome/nfp/nfp_devlink.c  |  55 +++----
 drivers/net/ethernet/netronome/nfp/nfp_main.c |  19 +--
 drivers/net/ethernet/netronome/nfp/nfp_main.h |   6 +-
 .../net/ethernet/netronome/nfp/nfp_net_main.c |  34 ++--
 .../net/ethernet/netronome/nfp/nfp_net_repr.c |   4 +-
 drivers/net/ethernet/netronome/nfp/nfp_port.c |  17 --
 drivers/net/ethernet/netronome/nfp/nfp_port.h |   2 -
 include/net/devlink.h                         |  15 +-
 net/core/devlink.c                            | 148 ++++++++++--------
 15 files changed, 196 insertions(+), 171 deletions(-)
Signature · 1 line
2.34.1
author maintainer
#1 lore raw

PATCH 1/6 devlink: expose instance locking and add locked port registering

It should be familiar and beneficial to expose devlink instance lock to the drivers. This way drivers can block devlink from calling them during critical sections without breakneck locking.

Add port helpers, port splitting callbacks will be the first target.

Use 'devl_' prefix for "explicitly locked" API. Initial RFC used '__devlink' but that's too much typing.

---
 include/net/devlink.h | 11 +++++
 net/core/devlink.c    | 99 ++++++++++++++++++++++++++++++++-----------
 2 files changed, 86 insertions(+), 24 deletions(-)
include/net/devlink.h +11 -0
index 8d5349d2fb68..9de0d091aee9 100644
@@ -1479,6 +1479,17 @@ void *devlink_priv(struct devlink *devlink);
 struct devlink *priv_to_devlink(void *priv);
 struct device *devlink_to_dev(const struct devlink *devlink);
 
+/* Devlink instance explicit locking */
+void devl_lock(struct devlink *devlink);
+void devl_unlock(struct devlink *devlink);
+void devl_assert_locked(struct devlink *devlink);
+bool devl_lock_is_held(struct devlink *devlink);
+
+int devl_port_register(struct devlink *devlink,
+		       struct devlink_port *devlink_port,
+		       unsigned int port_index);
+void devl_port_unregister(struct devlink_port *devlink_port);
+
 struct ib_device;
 
 struct net *devlink_net(const struct devlink *devlink);
net/core/devlink.c +75 -24

130 lines not on this page · raw message

Signature · 1 line
2.34.1
maintainer
#2 lore raw

Re: [RFT net-next 1/6] devlink: expose instance locking and add locked port registering

Quoted 21 lines · Thu, Mar 10, 2022 at 01:16:27AM CET, [email protected] wrote:

It should be familiar and beneficial to expose devlink instance lock to the drivers. This way drivers can block devlink from calling them during critical sections without breakneck locking.

Add port helpers, port splitting callbacks will be the first target.

Use 'devl_' prefix for "explicitly locked" API. Initial RFC used '__devlink' but that's too much typing.

---
include/net/devlink.h | 11 +++++
net/core/devlink.c | 99 ++++++++++++++++++++++++++++++++-----------

2 files changed, 86 insertions(+), 24 deletions(-)
include/net/devlink.h +0 -0
index 8d5349d2fb68..9de0d091aee9 100644

include/net/devlink.h void *devlink_priv(struct devlink *devlink); · line 1488

Quoted 10 lines
include/net/devlink.h +7 -0
@@ -1479,6 +1479,17 @@ void *devlink_priv(struct devlink *devlink);
 struct devlink *priv_to_devlink(void *priv);
 struct device *devlink_to_dev(const struct devlink *devlink);
 
+/* Devlink instance explicit locking */
+void devl_lock(struct devlink *devlink);
+void devl_unlock(struct devlink *devlink);
+void devl_assert_locked(struct devlink *devlink);
+bool devl_lock_is_held(struct devlink *devlink);
+
+int devl_port_register(struct devlink *devlink,

It is kind of confusing to have:
devlink_* - locked api
devl_* - unlocked api

And not really, because by this division, devl_lock() should be called devlink_lock(). So it is oddly mixed..

I believe that "_" or "__" prefix is prefered here and everyone knows with away what it it is good for.

If you find "__devlink_port_register" as "too much typing" (I don't), why don't we have all devlink api shortened to:
devl_*
and then the unlocked api could be called:
__devl_*
?

[...]

include/net/devlink.h void *devlink_priv(struct devlink *devlink); · line 1492

Quoted 4 lines
include/net/devlink.h +4 -0
+bool devl_lock_is_held(struct devlink *devlink)
+{
+	/* We have to check this at runtime because struct devlink
+	 * is now private. Normally lock_is_held() should be eliminated

"is now private" belong more to the patch description, not to the actual code I believe.

[...]

author maintainer
#3 lore raw

Re: [RFT net-next 1/6] devlink: expose instance locking and add locked port registering

Quoted 10 lines · On Thu, 10 Mar 2022 10:14:26 +0100 Jiri Pirko wrote:

It is kind of confusing to have:
devlink_* - locked api
devl_* - unlocked api

And not really, because by this division, devl_lock() should be called devlink_lock(). So it is oddly mixed..

I believe that "_" or "__" prefix is prefered here and everyone knows with away what it it is good for.

Quoted 6 lines

If you find "__devlink_port_register" as "too much typing" (I don't), why don't we have all devlink api shortened to:
devl_*
and then the unlocked api could be called:
__devl_*
?

The goal is for that API to be the main one, we can rename the devlink_ to something else at the end. The parts of it which are not completely removed.

Quoted 7 lines

net/core/devlink.c struct devlink *__must_check devlink_try_get(struct devlink *devlink) · line 237

Quoted 4 lines
net/core/devlink.c +4 -0
+bool devl_lock_is_held(struct devlink *devlink)
+{
+	/* We have to check this at runtime because struct devlink
+	 * is now private. Normally lock_is_held() should be eliminated  

"is now private" belong more to the patch description, not to the actual code I believe.

Alright. The comment started as a warning not to use this for anything but lockdep but I couldn't resist taking a dig at hiding the structure.

maintainer
#4 lore raw

Re: [RFT net-next 1/6] devlink: expose instance locking and add locked port registering

Quoted 18 lines · Thu, Mar 10, 2022 at 09:06:24PM CET, [email protected] wrote:
Quoted 10 lines · On Thu, 10 Mar 2022 10:14:26 +0100 Jiri Pirko wrote:

It is kind of confusing to have:
devlink_* - locked api
devl_* - unlocked api

And not really, because by this division, devl_lock() should be called devlink_lock(). So it is oddly mixed..

I believe that "_" or "__" prefix is prefered here and everyone knows with away what it it is good for.

Quoted 6 lines

If you find "__devlink_port_register" as "too much typing" (I don't), why don't we have all devlink api shortened to:
devl_*
and then the unlocked api could be called:
__devl_*
?

Quoted 3 lines

The goal is for that API to be the main one, we can rename the devlink_ to something else at the end. The parts of it which are not completely removed.

Okay. So please have it as:
devl_* - normal
__devl_* - unlocked

Thanks!

Quoted 11 lines
Quoted 8 lines

net/core/devlink.c struct devlink *__must_check devlink_try_get(struct devlink *devlink) · line 237

Quoted 5 lines
net/core/devlink.c +4 -0
+bool devl_lock_is_held(struct devlink *devlink)
+{
+	/* We have to check this at runtime because struct devlink
+	 * is now private. Normally lock_is_held() should be eliminated  

"is now private" belong more to the patch description, not to the actual code I believe.

Alright. The comment started as a warning not to use this for anything but lockdep but I couldn't resist taking a dig at hiding the structure.

author maintainer
#5 lore raw

Re: [RFT net-next 1/6] devlink: expose instance locking and add locked port registering

Quoted 7 lines · On Fri, 11 Mar 2022 10:15:30 +0100 Jiri Pirko wrote:
Quoted 3 lines

The goal is for that API to be the main one, we can rename the devlink_ to something else at the end. The parts of it which are not completely removed.

Okay. So please have it as:
devl_* - normal
__devl_* - unlocked

Isn't it fairly awkward for the main intended API to have __ in the name? __ means unsafe / make sure you know what you're doing.

There's little room for confusion here, we have locking asserts everywhere.

maintainer
#6 lore raw

Re: [RFT net-next 1/6] devlink: expose instance locking and add locked port registering

Quoted 9 lines · Fri, Mar 11, 2022 at 05:33:32PM CET, [email protected] wrote:
Quoted 7 lines · On Fri, 11 Mar 2022 10:15:30 +0100 Jiri Pirko wrote:
Quoted 3 lines

The goal is for that API to be the main one, we can rename the devlink_ to something else at the end. The parts of it which are not completely removed.

Okay. So please have it as:
devl_* - normal
__devl_* - unlocked

Quoted 5 lines

Isn't it fairly awkward for the main intended API to have __ in the name? __ means unsafe / make sure you know what you're doing.

There's little room for confusion here, we have locking asserts everywhere.

Well, I think it is common to have "__" prefixed for unlocked variant. Idk.

#7 lore raw

Re: [RFT net-next 1/6] devlink: expose instance locking and add locked port registering

In reply to Jakub Kicinski
Quoted 54 lines · Jakub Kicinski

It should be familiar and beneficial to expose devlink instance lock to the drivers. This way drivers can block devlink from calling them during critical sections without breakneck locking.

Add port helpers, port splitting callbacks will be the first target.

Use 'devl_' prefix for "explicitly locked" API. Initial RFC used '__devlink' but that's too much typing.

---
 include/net/devlink.h | 11 +++++
 net/core/devlink.c    | 99 ++++++++++++++++++++++++++++++++-----------
 2 files changed, 86 insertions(+), 24 deletions(-)
include/net/devlink.h +11 -0
index 8d5349d2fb68..9de0d091aee9 100644
@@ -1479,6 +1479,17 @@ void *devlink_priv(struct devlink *devlink);
 struct devlink *priv_to_devlink(void *priv);
 struct device *devlink_to_dev(const struct devlink *devlink);
 
+/* Devlink instance explicit locking */
+void devl_lock(struct devlink *devlink);
+void devl_unlock(struct devlink *devlink);
+void devl_assert_locked(struct devlink *devlink);
+bool devl_lock_is_held(struct devlink *devlink);
+
+int devl_port_register(struct devlink *devlink,
+		       struct devlink_port *devlink_port,
+		       unsigned int port_index);
+void devl_port_unregister(struct devlink_port *devlink_port);
+
 struct ib_device;
 
 struct net *devlink_net(const struct devlink *devlink);
net/core/devlink.c +8 -0
index fcd9f6d85cf1..c30da1fc023d 100644
@@ -225,6 +225,37 @@ struct devlink *__must_check devlink_try_get(struct devlink *devlink)
 	return NULL;
 }
 
+void devl_assert_locked(struct devlink *devlink)
+{
+	lockdep_assert_held(&devlink->lock);
+}
+EXPORT_SYMBOL_GPL(devl_assert_locked);
+
+bool devl_lock_is_held(struct devlink *devlink)
+{

net/core/devlink.c struct devlink *__must_check devlink_try_get(struct devlink *devlink) · line 245

Quoted 10 lines
net/core/devlink.c +10 -0
+	/* We have to check this at runtime because struct devlink
+	 * is now private. Normally lock_is_held() should be eliminated
+	 * as dead code in the caller and we can depend on the linker error.
+	 */
+	if (!IS_ENABLED(CONFIG_LOCKDEP))
+		return WARN_ON_ONCE(true);
+
+	return lockdep_is_held(&devlink->lock);
+}
+EXPORT_SYMBOL_GPL(devl_lock_is_held);

What about this?

include/net/devlink.h +8 -1
index 8d5349d2fb68..33b47d1a6800 100644
@@ -1762,5 +1762,12 @@ devlink_compat_switch_id_get(struct net_device *dev,
 }
 
 #endif
-
+#if IS_ENABLED(CONFIG_LOCKDEP)
+bool devl_lock_is_held(struct devlink *devlink);
+#else
+static inline bool devl_lock_is_held(struct devlink *devlink)
+{
+       return true;
+}
+#endif
 #endif /* _NET_DEVLINK_H_ */
author maintainer
#8 lore raw

Re: [RFT net-next 1/6] devlink: expose instance locking and add locked port registering

Quoted 1 line · On Fri, 11 Mar 2022 18:09:36 +0200 Leon Romanovsky wrote:

What about this?

Is it better? Can do it you prefer, but I'd lean towards a version without an ifdef myself.

Quoted 18 lines
include/net/devlink.h +8 -1
index 8d5349d2fb68..33b47d1a6800 100644
@@ -1762,5 +1762,12 @@ devlink_compat_switch_id_get(struct net_device *dev,
 }
 
 #endif
-
+#if IS_ENABLED(CONFIG_LOCKDEP)
+bool devl_lock_is_held(struct devlink *devlink);
+#else
+static inline bool devl_lock_is_held(struct devlink *devlink)
+{
+       return true;
+}
+#endif
 #endif /* _NET_DEVLINK_H_ */
#9 lore raw

Re: [RFT net-next 1/6] devlink: expose instance locking and add locked port registering

Quoted 4 lines · Jakub Kicinski
Quoted 1 line · On Fri, 11 Mar 2022 18:09:36 +0200 Leon Romanovsky wrote:

What about this?

Is it better?

I think so. It doesn't create shadow dependency on LOCKDEP. In your variant, all users of this call will generate WARN in production systems that run without lockdep.

So if you want the "eliminate" thing like you wrote in the comment, the ifdef is a common solution.

Quoted 2 lines

Can do it you prefer, but I'd lean towards a version without an ifdef myself.

So you need to add CONFIG_LOCKDEP dependency in devlink Kconfig.

Thanks

Quoted 19 lines
Quoted 19 lines
include/net/devlink.h +8 -1
index 8d5349d2fb68..33b47d1a6800 100644
@@ -1762,5 +1762,12 @@ devlink_compat_switch_id_get(struct net_device *dev,
 }
 
 #endif
-
+#if IS_ENABLED(CONFIG_LOCKDEP)
+bool devl_lock_is_held(struct devlink *devlink);
+#else
+static inline bool devl_lock_is_held(struct devlink *devlink)
+{
+       return true;
+}
+#endif
 #endif /* _NET_DEVLINK_H_ */
author maintainer
#10 lore raw

Re: [RFT net-next 1/6] devlink: expose instance locking and add locked port registering

Quoted 9 lines · On Fri, 11 Mar 2022 18:57:35 +0200 Leon Romanovsky wrote:
Quoted 4 lines · Jakub Kicinski
Quoted 1 line · On Fri, 11 Mar 2022 18:09:36 +0200 Leon Romanovsky wrote:

What about this?

Is it better?

I think so. It doesn't create shadow dependency on LOCKDEP. In your variant, all users of this call will generate WARN in production systems that run without lockdep.

No, no, that function is mostly for rcu dereference checking. The calls should be eliminated as dead code on production systems.

Quoted 2 lines

So if you want the "eliminate" thing like you wrote in the comment, the ifdef is a common solution.

I think these days people try to use IS_ENABLED() whenever possible.

Quoted 4 lines
Quoted 2 lines

Can do it you prefer, but I'd lean towards a version without an ifdef myself.

So you need to add CONFIG_LOCKDEP dependency in devlink Kconfig.

I don't see why.

author maintainer
#11 lore raw

Re: [RFT net-next 1/6] devlink: expose instance locking and add locked port registering

Quoted 11 lines · On Fri, 11 Mar 2022 09:39:13 -0800 Jakub Kicinski wrote:
Quoted 3 lines

I think so. It doesn't create shadow dependency on LOCKDEP. In your variant, all users of this call will generate WARN in production systems that run without lockdep.

No, no, that function is mostly for rcu dereference checking. The calls should be eliminated as dead code on production systems.

Quoted 2 lines

So if you want the "eliminate" thing like you wrote in the comment, the ifdef is a common solution.

I think these days people try to use IS_ENABLED() whenever possible.

Let me wrap the implementation in an ifdef, I guess that may be least confusing and the linker error vs runtime warning is worth the cost.

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

Can do it you prefer, but I'd lean towards a version without an ifdef myself.

So you need to add CONFIG_LOCKDEP dependency in devlink Kconfig.

I don't see why.

#12 lore raw

Re: [RFT net-next 1/6] devlink: expose instance locking and add locked port registering

In reply to Jakub Kicinski
Quoted 5 lines · Jakub Kicinski
Quoted 4 lines · On Fri, 11 Mar 2022 18:57:35 +0200 Leon Romanovsky wrote:
Quoted 3 lines · Jakub Kicinski
Quoted 1 line · On Fri, 11 Mar 2022 18:09:36 +0200 Leon Romanovsky wrote:

What about this?

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

Is it better?

I think so. It doesn't create shadow dependency on LOCKDEP. In your variant, all users of this call will generate WARN in production systems that run without lockdep.

No, no, that function is mostly for rcu dereference checking. The calls should be eliminated as dead code on production systems.

On systems without LOCKDEP, the devl_lock_is_held function will be generated to be like this:
bool devl_lock_is_held(struct devlink *devlink)

{
	return WARN_ON_ONCE(true);
}

EXPORT_SYMBOL_GPL(devl_lock_is_held);

Quoted 5 lines
Quoted 3 lines

So if you want the "eliminate" thing like you wrote in the comment, the ifdef is a common solution.

I think these days people try to use IS_ENABLED() whenever possible.

I'm one of such people, but here you put always WARN if LOCKDEP is not enabled.

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

Can do it you prefer, but I'd lean towards a version without an ifdef myself.

So you need to add CONFIG_LOCKDEP dependency in devlink Kconfig.

I don't see why.

Because of WARN_ON_ONCE.

Thanks

author maintainer
#13 lore raw

Re: [RFT net-next 1/6] devlink: expose instance locking and add locked port registering

Quoted 10 lines · On Fri, 11 Mar 2022 19:49:36 +0200 Leon Romanovsky wrote:
Quoted 2 lines

No, no, that function is mostly for rcu dereference checking. The calls should be eliminated as dead code on production systems.

On systems without LOCKDEP, the devl_lock_is_held function will be generated to be like this:
bool devl_lock_is_held(struct devlink *devlink)

{
	return WARN_ON_ONCE(true);
}

EXPORT_SYMBOL_GPL(devl_lock_is_held);

I think you missed my first sentence. Anyway, this is what I'll do in v1:

#ifdef CONFIG_LOCKDEP
/* For use in conjunction with LOCKDEP only e.g. rcu_dereference_protected() */ bool devl_lock_is_held(struct devlink *devlink)

{
	return lockdep_is_held(&devlink->lock);
}

EXPORT_SYMBOL_GPL(devl_lock_is_held);
#endif

#14 lore raw

Re: [RFT net-next 1/6] devlink: expose instance locking and add locked port registering

Quoted 15 lines · Jakub Kicinski
Quoted 10 lines · On Fri, 11 Mar 2022 19:49:36 +0200 Leon Romanovsky wrote:
Quoted 2 lines

No, no, that function is mostly for rcu dereference checking. The calls should be eliminated as dead code on production systems.

On systems without LOCKDEP, the devl_lock_is_held function will be generated to be like this:
bool devl_lock_is_held(struct devlink *devlink)

{
	return WARN_ON_ONCE(true);
}

EXPORT_SYMBOL_GPL(devl_lock_is_held);

I think you missed my first sentence. Anyway, this is what I'll do in v1:

Quoted 8 lines

#ifdef CONFIG_LOCKDEP
/* For use in conjunction with LOCKDEP only e.g. rcu_dereference_protected() */ bool devl_lock_is_held(struct devlink *devlink)

{
	return lockdep_is_held(&devlink->lock);
}

EXPORT_SYMBOL_GPL(devl_lock_is_held);
#endif

Because you put EXPORT_SYMBOL_GPL, this function will be used by drivers too, which will need to write something like this:

if (IS_ENABLED(CONFIG_LOCKDEP))
devl_lock_is_held(devlink)

Otherwise, they will get linkage error in systems without LOCKDEP.

Thanks

author maintainer
#15 lore raw

PATCH 2/6 eth: nfp: wrap locking assertions in helpers

In reply to Jakub Kicinski

We can replace the PF lock with devlink instance lock in subsequent changes. To make the patches easier to comprehend and limit line lengths - factor out the existing locking assertions.

No functional changes.

---
 drivers/net/ethernet/netronome/nfp/flower/main.c  | 4 ++--
 drivers/net/ethernet/netronome/nfp/nfp_app.c      | 2 +-
 drivers/net/ethernet/netronome/nfp/nfp_app.h      | 9 +++++++++
 drivers/net/ethernet/netronome/nfp/nfp_net_repr.c | 4 ++--
 4 files changed, 14 insertions(+), 5 deletions(-)
drivers/net/ethernet/netronome/nfp/flower/main.c +2 -2
index ac1dcfa1d179..4d960a9641b3 100644
@@ -266,7 +266,7 @@ nfp_flower_reprs_reify(struct nfp_app *app, enum nfp_repr_type type,
 	int i, err, count = 0;
 
 	reprs = rcu_dereference_protected(app->reprs[type],
-					  lockdep_is_held(&app->pf->lock));
+					  nfp_app_is_locked(app));
 	if (!reprs)
 		return 0;
 
@@ -295,7 +295,7 @@ nfp_flower_wait_repr_reify(struct nfp_app *app, atomic_t *replies, int tot_repl)
 	if (!tot_repl)
 		return 0;
 
-	lockdep_assert_held(&app->pf->lock);
+	assert_nfp_app_locked(app);
 	if (!wait_event_timeout(priv->reify_wait_queue,
 				atomic_read(replies) >= tot_repl,
 				NFP_FL_REPLY_TIMEOUT)) {
drivers/net/ethernet/netronome/nfp/nfp_app.c +1 -1
index 3a973282b2bb..09f250e74dfa 100644
@@ -121,7 +121,7 @@ struct nfp_reprs *
 nfp_reprs_get_locked(struct nfp_app *app, enum nfp_repr_type type)
 {
 	return rcu_dereference_protected(app->reprs[type],
-					 lockdep_is_held(&app->pf->lock));
+					 nfp_app_is_locked(app));
 }
 
 struct nfp_reprs *
drivers/net/ethernet/netronome/nfp/nfp_app.h +9 -0
index 3e9baff07100..60cb8a71e02d 100644
@@ -4,10 +4,12 @@
 #ifndef _NFP_APP_H
 #define _NFP_APP_H 1
 
+#include <linux/lockdep.h>
 #include <net/devlink.h>
 
 #include <trace/events/devlink.h>
 
+#include "nfp_main.h"
 #include "nfp_net_repr.h"
 
 #define NFP_APP_CTRL_MTU_MAX	U32_MAX
@@ -174,6 +176,13 @@ struct nfp_app {
 	void *priv;
 };
 
+static inline void assert_nfp_app_locked(struct nfp_app *app)
+{
+	lockdep_assert_held(&app->pf->lock);
+}
+
+#define nfp_app_is_locked(app)	lockdep_is_held(&(app)->pf->lock)
+
 void nfp_check_rhashtable_empty(void *ptr, void *arg);
 bool __nfp_ctrl_tx(struct nfp_net *nn, struct sk_buff *skb);
 bool nfp_ctrl_tx(struct nfp_net *nn, struct sk_buff *skb);
drivers/net/ethernet/netronome/nfp/nfp_net_repr.c +2 -2
index 181ac8e789a3..ba3fa7eac98d 100644
@@ -20,7 +20,7 @@ struct net_device *
 nfp_repr_get_locked(struct nfp_app *app, struct nfp_reprs *set, unsigned int id)
 {
 	return rcu_dereference_protected(set->reprs[id],
-					 lockdep_is_held(&app->pf->lock));
+					 nfp_app_is_locked(app));
 }
 
 static void
@@ -476,7 +476,7 @@ nfp_reprs_clean_and_free_by_type(struct nfp_app *app, enum nfp_repr_type type)
 	int i;
 
 	reprs = rcu_dereference_protected(app->reprs[type],
-					  lockdep_is_held(&app->pf->lock));
+					  nfp_app_is_locked(app));
 	if (!reprs)
 		return;
 
Signature · 1 line
2.34.1
author maintainer
#16 lore raw

PATCH 3/6 eth: nfp: replace driver's "pf" lock with devlink instance lock

In reply to Jakub Kicinski

The whole reason for existence of the pf mutex is that we could not lock the devlink instance around port splitting. There are more types of reconfig which can make ports appear or disappear. Now that the devlink instance lock is exposed to drivers and "locked" helpers exist we can switch to using the devlink lock directly.

Next patches will move the locking inside .port_(un)split to the core.

---
 drivers/net/ethernet/netronome/nfp/nfp_app.h  | 11 +++---
 .../net/ethernet/netronome/nfp/nfp_devlink.c  | 16 ++++-----
 drivers/net/ethernet/netronome/nfp/nfp_main.c | 19 ++++++-----
 drivers/net/ethernet/netronome/nfp/nfp_main.h |  6 ++--
 .../net/ethernet/netronome/nfp/nfp_net_main.c | 34 +++++++++++--------
 drivers/net/ethernet/netronome/nfp/nfp_port.c |  3 +-
 6 files changed, 48 insertions(+), 41 deletions(-)
drivers/net/ethernet/netronome/nfp/nfp_app.h +6 -5
index 60cb8a71e02d..dd56207df246 100644
@@ -4,12 +4,10 @@
 #ifndef _NFP_APP_H
 #define _NFP_APP_H 1
 
-#include <linux/lockdep.h>
 #include <net/devlink.h>
 
 #include <trace/events/devlink.h>
 
-#include "nfp_main.h"
 #include "nfp_net_repr.h"
 
 #define NFP_APP_CTRL_MTU_MAX	U32_MAX
@@ -77,7 +75,7 @@ extern const struct nfp_app_type app_abm;
  * @bpf:	BPF ndo offload-related calls
  * @xdp_offload:    offload an XDP program
  * @eswitch_mode_get:    get SR-IOV eswitch mode
- * @eswitch_mode_set:    set SR-IOV eswitch mode (under pf->lock)
+ * @eswitch_mode_set:    set SR-IOV eswitch mode
  * @sriov_enable: app-specific sriov initialisation
  * @sriov_disable: app-specific sriov clean-up
  * @dev_get:	get representor or internal port representing netdev
@@ -178,10 +176,13 @@ struct nfp_app {
 
 static inline void assert_nfp_app_locked(struct nfp_app *app)
 {
-	lockdep_assert_held(&app->pf->lock);
+	devl_assert_locked(priv_to_devlink(app->pf));
 }
 
-#define nfp_app_is_locked(app)	lockdep_is_held(&(app)->pf->lock)
+static inline bool nfp_app_is_locked(struct nfp_app *app)
+{
+	return devl_lock_is_held(priv_to_devlink(app->pf));
+}
 
 void nfp_check_rhashtable_empty(void *ptr, void *arg);
 bool __nfp_ctrl_tx(struct nfp_net *nn, struct sk_buff *skb);
drivers/net/ethernet/netronome/nfp/nfp_devlink.c +8 -8
index bea978df7713..865f62958a72 100644
@@ -70,7 +70,7 @@ nfp_devlink_port_split(struct devlink *devlink, unsigned int port_index,
 	unsigned int lanes;
 	int ret;
 
-	mutex_lock(&pf->lock);
+	devl_lock(devlink);
 
 	rtnl_lock();
 	ret = nfp_devlink_fill_eth_port_from_id(pf, port_index, &eth_port);
@@ -90,7 +90,7 @@ nfp_devlink_port_split(struct devlink *devlink, unsigned int port_index,
 
 	ret = nfp_devlink_set_lanes(pf, eth_port.index, lanes);
 out:
-	mutex_unlock(&pf->lock);
+	devl_unlock(devlink);
 
 	return ret;
 }
@@ -104,7 +104,7 @@ nfp_devlink_port_unsplit(struct devlink *devlink, unsigned int port_index,
 	unsigned int lanes;
 	int ret;
 
-	mutex_lock(&pf->lock);
+	devl_lock(devlink);
 
 	rtnl_lock();
 	ret = nfp_devlink_fill_eth_port_from_id(pf, port_index, &eth_port);
@@ -124,7 +124,7 @@ nfp_devlink_port_unsplit(struct devlink *devlink, unsigned int port_index,
 
 	ret = nfp_devlink_set_lanes(pf, eth_port.index, lanes);
 out:
-	mutex_unlock(&pf->lock);
+	devl_unlock(devlink);
 
 	return ret;
 }
@@ -163,9 +163,9 @@ static int nfp_devlink_eswitch_mode_set(struct devlink *devlink, u16 mode,
 	struct nfp_pf *pf = devlink_priv(devlink);
 	int ret;
 
-	mutex_lock(&pf->lock);
+	devl_lock(devlink);
 	ret = nfp_app_eswitch_mode_set(pf->app, mode);
-	mutex_unlock(&pf->lock);
+	devl_unlock(devlink);
 
 	return ret;
 }
@@ -375,12 +375,12 @@ int nfp_devlink_port_register(struct nfp_app *app, struct nfp_port *port)
 
 	devlink = priv_to_devlink(app->pf);
 
-	return devlink_port_register(devlink, &port->dl_port, port->eth_id);
+	return devl_port_register(devlink, &port->dl_port, port->eth_id);
 }
 
 void nfp_devlink_port_unregister(struct nfp_port *port)
 {
-	devlink_port_unregister(&port->dl_port);
+	devl_port_unregister(&port->dl_port);
 }
 
 void nfp_devlink_port_type_eth_set(struct nfp_port *port)
drivers/net/ethernet/netronome/nfp/nfp_main.c +10 -9
index bb3b8a7f6c5d..a42068d706cb 100644
@@ -222,6 +222,7 @@ static int nfp_pcie_sriov_enable(struct pci_dev *pdev, int num_vfs)
 {
 #ifdef CONFIG_PCI_IOV
 	struct nfp_pf *pf = pci_get_drvdata(pdev);
+	struct devlink *devlink;
 	int err;
 
 	if (num_vfs > pf->limit_vfs) {
@@ -236,7 +237,8 @@ static int nfp_pcie_sriov_enable(struct pci_dev *pdev, int num_vfs)
 		return err;
 	}
 
-	mutex_lock(&pf->lock);
+	devlink = priv_to_devlink(pf);
+	devl_lock(devlink);
 
 	err = nfp_app_sriov_enable(pf->app, num_vfs);
 	if (err) {
@@ -250,11 +252,11 @@ static int nfp_pcie_sriov_enable(struct pci_dev *pdev, int num_vfs)
 
 	dev_dbg(&pdev->dev, "Created %d VFs.\n", pf->num_vfs);
 
-	mutex_unlock(&pf->lock);
+	devl_unlock(devlink);
 	return num_vfs;
 
 err_sriov_disable:
-	mutex_unlock(&pf->lock);
+	devl_unlock(devlink);
 	pci_disable_sriov(pdev);
 	return err;
 #endif
@@ -265,8 +267,10 @@ static int nfp_pcie_sriov_disable(struct pci_dev *pdev)
 {
 #ifdef CONFIG_PCI_IOV
 	struct nfp_pf *pf = pci_get_drvdata(pdev);
+	struct devlink *devlink;
 
-	mutex_lock(&pf->lock);
+	devlink = priv_to_devlink(pf);
+	devl_lock(devlink);
 
 	/* If the VFs are assigned we cannot shut down SR-IOV without
 	 * causing issues, so just leave the hardware available but
@@ -274,7 +278,7 @@ static int nfp_pcie_sriov_disable(struct pci_dev *pdev)
 	 */
 	if (pci_vfs_assigned(pdev)) {
 		dev_warn(&pdev->dev, "Disabling while VFs assigned - VFs will not be deallocated\n");
-		mutex_unlock(&pf->lock);
+		devl_unlock(devlink);
 		return -EPERM;
 	}
 
@@ -282,7 +286,7 @@ static int nfp_pcie_sriov_disable(struct pci_dev *pdev)
 
 	pf->num_vfs = 0;
 
-	mutex_unlock(&pf->lock);
+	devl_unlock(devlink);
 
 	pci_disable_sriov(pdev);
 	dev_dbg(&pdev->dev, "Removed VFs.\n");
@@ -700,7 +704,6 @@ static int nfp_pci_probe(struct pci_dev *pdev,
 	pf = devlink_priv(devlink);
 	INIT_LIST_HEAD(&pf->vnics);
 	INIT_LIST_HEAD(&pf->ports);
-	mutex_init(&pf->lock);
 	pci_set_drvdata(pdev, pf);
 	pf->pdev = pdev;
 
@@ -790,7 +793,6 @@ static int nfp_pci_probe(struct pci_dev *pdev,
 	destroy_workqueue(pf->wq);
 err_pci_priv_unset:
 	pci_set_drvdata(pdev, NULL);
-	mutex_destroy(&pf->lock);
 	devlink_free(devlink);
 err_rel_regions:
 	pci_release_regions(pdev);
@@ -827,7 +829,6 @@ static void __nfp_pci_shutdown(struct pci_dev *pdev, bool unload_fw)
 
 	kfree(pf->eth_tbl);
 	kfree(pf->nspi);
-	mutex_destroy(&pf->lock);
 	devlink_free(priv_to_devlink(pf));
 	pci_release_regions(pdev);
 	pci_disable_device(pdev);
drivers/net/ethernet/netronome/nfp/nfp_main.h +2 -4
index a7dede946a33..20129a6aaea0 100644
@@ -13,7 +13,6 @@
 #include <linux/list.h>
 #include <linux/types.h>
 #include <linux/msi.h>
-#include <linux/mutex.h>
 #include <linux/pci.h>
 #include <linux/workqueue.h>
 #include <net/devlink.h>
@@ -84,7 +83,8 @@ struct nfp_dumpspec {
  * @port_refresh_work:	Work entry for taking netdevs out
  * @shared_bufs:	Array of shared buffer structures if FW has any SBs
  * @num_shared_bufs:	Number of elements in @shared_bufs
- * @lock:		Protects all fields which may change after probe
+ *
+ * Fields which may change after proble are protected by devlink instance lock.
  */
 struct nfp_pf {
 	struct pci_dev *pdev;
@@ -139,8 +139,6 @@ struct nfp_pf {
 
 	struct nfp_shared_buf *shared_bufs;
 	unsigned int num_shared_bufs;
-
-	struct mutex lock;
 };
 
 extern struct pci_driver nfp_netvf_pci_driver;
drivers/net/ethernet/netronome/nfp/nfp_net_main.c +20 -14
index 751f76cd4f79..efcc76a06a9a 100644
@@ -307,6 +307,7 @@ static int nfp_net_pf_init_vnics(struct nfp_pf *pf)
 static int
 nfp_net_pf_app_init(struct nfp_pf *pf, u8 __iomem *qc_bar, unsigned int stride)
 {
+	struct devlink *devlink = priv_to_devlink(pf);
 	u8 __iomem *ctrl_bar;
 	int err;
 
@@ -314,9 +315,9 @@ nfp_net_pf_app_init(struct nfp_pf *pf, u8 __iomem *qc_bar, unsigned int stride)
 	if (IS_ERR(pf->app))
 		return PTR_ERR(pf->app);
 
-	mutex_lock(&pf->lock);
+	devl_lock(devlink);
 	err = nfp_app_init(pf->app);
-	mutex_unlock(&pf->lock);
+	devl_unlock(devlink);
 	if (err)
 		goto err_free;
 
@@ -343,9 +344,9 @@ nfp_net_pf_app_init(struct nfp_pf *pf, u8 __iomem *qc_bar, unsigned int stride)
 err_unmap:
 	nfp_cpp_area_release_free(pf->ctrl_vnic_bar);
 err_app_clean:
-	mutex_lock(&pf->lock);
+	devl_lock(devlink);
 	nfp_app_clean(pf->app);
-	mutex_unlock(&pf->lock);
+	devl_unlock(devlink);
 err_free:
 	nfp_app_free(pf->app);
 	pf->app = NULL;
@@ -354,14 +355,16 @@ nfp_net_pf_app_init(struct nfp_pf *pf, u8 __iomem *qc_bar, unsigned int stride)
 
 static void nfp_net_pf_app_clean(struct nfp_pf *pf)
 {
+	struct devlink *devlink = priv_to_devlink(pf);
+
 	if (pf->ctrl_vnic) {
 		nfp_net_pf_free_vnic(pf, pf->ctrl_vnic);
 		nfp_cpp_area_release_free(pf->ctrl_vnic_bar);
 	}
 
-	mutex_lock(&pf->lock);
+	devl_lock(devlink);
 	nfp_app_clean(pf->app);
-	mutex_unlock(&pf->lock);
+	devl_unlock(devlink);
 
 	nfp_app_free(pf->app);
 	pf->app = NULL;
@@ -546,12 +549,13 @@ nfp_net_eth_port_update(struct nfp_cpp *cpp, struct nfp_port *port,
 
 int nfp_net_refresh_port_table_sync(struct nfp_pf *pf)
 {
+	struct devlink *devlink = priv_to_devlink(pf);
 	struct nfp_eth_table *eth_table;
 	struct nfp_net *nn, *next;
 	struct nfp_port *port;
 	int err;
 
-	lockdep_assert_held(&pf->lock);
+	devl_assert_locked(devlink);
 
 	/* Check for nfp_net_pci_remove() racing against us */
 	if (list_empty(&pf->vnics))
@@ -600,10 +604,11 @@ static void nfp_net_refresh_vnics(struct work_struct *work)
 {
 	struct nfp_pf *pf = container_of(work, struct nfp_pf,
 					 port_refresh_work);
+	struct devlink *devlink = priv_to_devlink(pf);
 
-	mutex_lock(&pf->lock);
+	devl_lock(devlink);
 	nfp_net_refresh_port_table_sync(pf);
-	mutex_unlock(&pf->lock);
+	devl_unlock(devlink);
 }
 
 void nfp_net_refresh_port_table(struct nfp_port *port)
@@ -709,7 +714,7 @@ int nfp_net_pci_probe(struct nfp_pf *pf)
 	if (err)
 		goto err_shared_buf_unreg;
 
-	mutex_lock(&pf->lock);
+	devl_lock(devlink);
 	pf->ddir = nfp_net_debugfs_device_add(pf->pdev);
 
 	/* Allocate the vnics and do basic init */
@@ -729,7 +734,7 @@ int nfp_net_pci_probe(struct nfp_pf *pf)
 	if (err)
 		goto err_stop_app;
 
-	mutex_unlock(&pf->lock);
+	devl_unlock(devlink);
 	devlink_register(devlink);
 
 	return 0;
@@ -742,7 +747,7 @@ int nfp_net_pci_probe(struct nfp_pf *pf)
 	nfp_net_pf_free_vnics(pf);
 err_clean_ddir:
 	nfp_net_debugfs_dir_clean(&pf->ddir);
-	mutex_unlock(&pf->lock);
+	devl_unlock(devlink);
 	nfp_devlink_params_unregister(pf);
 err_shared_buf_unreg:
 	nfp_shared_buf_unregister(pf);
@@ -756,10 +761,11 @@ int nfp_net_pci_probe(struct nfp_pf *pf)
 
 void nfp_net_pci_remove(struct nfp_pf *pf)
 {
+	struct devlink *devlink = priv_to_devlink(pf);
 	struct nfp_net *nn, *next;
 
 	devlink_unregister(priv_to_devlink(pf));
-	mutex_lock(&pf->lock);
+	devl_lock(devlink);
 	list_for_each_entry_safe(nn, next, &pf->vnics, vnic_list) {
 		if (!nfp_net_is_data_vnic(nn))
 			continue;
@@ -771,7 +777,7 @@ void nfp_net_pci_remove(struct nfp_pf *pf)
 	/* stop app first, to avoid double free of ctrl vNIC's ddir */
 	nfp_net_debugfs_dir_clean(&pf->ddir);
 
-	mutex_unlock(&pf->lock);
+	devl_unlock(devlink);
 
 	nfp_devlink_params_unregister(pf);
 	nfp_shared_buf_unregister(pf);
drivers/net/ethernet/netronome/nfp/nfp_port.c +2 -1
index 93c5bfc0510b..6f3ea1700a15 100644
@@ -78,9 +78,10 @@ int nfp_port_set_features(struct net_device *netdev, netdev_features_t features)
 struct nfp_port *
 nfp_port_from_id(struct nfp_pf *pf, enum nfp_port_type type, unsigned int id)
 {
+	struct devlink *devlink = priv_to_devlink(pf);
 	struct nfp_port *port;
 
-	lockdep_assert_held(&pf->lock);
+	assert_devl_locked(devlink);
 
 	if (type != NFP_PORT_PHYS_PORT)
 		return NULL;
Signature · 1 line
2.34.1
author maintainer
#17 lore raw

PATCH 4/6 eth: mlxsw: switch to explicit locking for port registration

In reply to Jakub Kicinski

Explicitly lock the devlink instance and use devl_ API.

This will be used by the subsequent patch to invoke .port_split / .port_unsplit callbacks with devlink instance lock held.

---
 drivers/net/ethernet/mellanox/mlxsw/core.c     |  4 ++--
 drivers/net/ethernet/mellanox/mlxsw/minimal.c  |  6 ++++++
 drivers/net/ethernet/mellanox/mlxsw/spectrum.c | 14 ++++++++++++++
 3 files changed, 22 insertions(+), 2 deletions(-)
drivers/net/ethernet/mellanox/mlxsw/core.c +2 -2
index 0bf1d64644ba..e2a6a759eb6c 100644
@@ -2983,7 +2983,7 @@ static int __mlxsw_core_port_init(struct mlxsw_core *mlxsw_core, u16 local_port,
 	attrs.switch_id.id_len = switch_id_len;
 	mlxsw_core_port->local_port = local_port;
 	devlink_port_attrs_set(devlink_port, &attrs);
-	err = devlink_port_register(devlink, devlink_port, local_port);
+	err = devl_port_register(devlink, devlink_port, local_port);
 	if (err)
 		memset(mlxsw_core_port, 0, sizeof(*mlxsw_core_port));
 	return err;
@@ -2995,7 +2995,7 @@ static void __mlxsw_core_port_fini(struct mlxsw_core *mlxsw_core, u16 local_port
 					&mlxsw_core->ports[local_port];
 	struct devlink_port *devlink_port = &mlxsw_core_port->devlink_port;
 
-	devlink_port_unregister(devlink_port);
+	devl_port_unregister(devlink_port);
 	memset(mlxsw_core_port, 0, sizeof(*mlxsw_core_port));
 }
 
drivers/net/ethernet/mellanox/mlxsw/minimal.c +6 -0
index 060209983438..3bc012dafd08 100644
@@ -422,6 +422,7 @@ static int mlxsw_m_init(struct mlxsw_core *mlxsw_core,
 			struct netlink_ext_ack *extack)
 {
 	struct mlxsw_m *mlxsw_m = mlxsw_core_driver_priv(mlxsw_core);
+	struct devlink *devlink = priv_to_devlink(mlxsw_core);
 	int err;
 
 	mlxsw_m->core = mlxsw_core;
@@ -437,7 +438,9 @@ static int mlxsw_m_init(struct mlxsw_core *mlxsw_core,
 		return err;
 	}
 
+	devl_lock(devlink);
 	err = mlxsw_m_ports_create(mlxsw_m);
+	devl_unlock(devlink);
 	if (err) {
 		dev_err(mlxsw_m->bus_info->dev, "Failed to create ports\n");
 		return err;
@@ -449,8 +452,11 @@ static int mlxsw_m_init(struct mlxsw_core *mlxsw_core,
 static void mlxsw_m_fini(struct mlxsw_core *mlxsw_core)
 {
 	struct mlxsw_m *mlxsw_m = mlxsw_core_driver_priv(mlxsw_core);
+	struct devlink *devlink = priv_to_devlink(mlxsw_core);
 
+	devl_lock(devlink);
 	mlxsw_m_ports_remove(mlxsw_m);
+	devl_unlock(devlink);
 }
 
 static const struct mlxsw_config_profile mlxsw_m_config_profile;
drivers/net/ethernet/mellanox/mlxsw/spectrum.c +14 -0
index 7b7b17183d10..1e823b669d1c 100644
@@ -2025,6 +2025,7 @@ static int mlxsw_sp_port_split(struct mlxsw_core *mlxsw_core, u16 local_port,
 			       struct netlink_ext_ack *extack)
 {
 	struct mlxsw_sp *mlxsw_sp = mlxsw_core_driver_priv(mlxsw_core);
+	struct devlink *devlink = priv_to_devlink(mlxsw_core);
 	struct mlxsw_sp_port_mapping port_mapping;
 	struct mlxsw_sp_port *mlxsw_sp_port;
 	enum mlxsw_reg_pmtdb_status status;
@@ -2062,6 +2063,7 @@ static int mlxsw_sp_port_split(struct mlxsw_core *mlxsw_core, u16 local_port,
 
 	port_mapping = mlxsw_sp_port->mapping;
 
+	devl_lock(devlink);
 	for (i = 0; i < count; i++) {
 		u16 s_local_port = mlxsw_reg_pmtdb_port_num_get(pmtdb_pl, i);
 
@@ -2075,11 +2077,13 @@ static int mlxsw_sp_port_split(struct mlxsw_core *mlxsw_core, u16 local_port,
 		dev_err(mlxsw_sp->bus_info->dev, "Failed to create split ports\n");
 		goto err_port_split_create;
 	}
+	devl_unlock(devlink);
 
 	return 0;
 
 err_port_split_create:
 	mlxsw_sp_port_unsplit_create(mlxsw_sp, count, pmtdb_pl);
+	devl_unlock(devlink);
 	return err;
 }
 
@@ -2087,6 +2091,7 @@ static int mlxsw_sp_port_unsplit(struct mlxsw_core *mlxsw_core, u16 local_port,
 				 struct netlink_ext_ack *extack)
 {
 	struct mlxsw_sp *mlxsw_sp = mlxsw_core_driver_priv(mlxsw_core);
+	struct devlink *devlink = priv_to_devlink(mlxsw_core);
 	struct mlxsw_sp_port *mlxsw_sp_port;
 	char pmtdb_pl[MLXSW_REG_PMTDB_LEN];
 	unsigned int count;
@@ -2118,6 +2123,7 @@ static int mlxsw_sp_port_unsplit(struct mlxsw_core *mlxsw_core, u16 local_port,
 		return err;
 	}
 
+	devl_lock(devlink);
 	for (i = 0; i < count; i++) {
 		u16 s_local_port = mlxsw_reg_pmtdb_port_num_get(pmtdb_pl, i);
 
@@ -2126,6 +2132,7 @@ static int mlxsw_sp_port_unsplit(struct mlxsw_core *mlxsw_core, u16 local_port,
 	}
 
 	mlxsw_sp_port_unsplit_create(mlxsw_sp, count, pmtdb_pl);
+	devl_unlock(devlink);
 
 	return 0;
 }
@@ -2818,6 +2825,7 @@ static int mlxsw_sp_init(struct mlxsw_core *mlxsw_core,
 			 struct netlink_ext_ack *extack)
 {
 	struct mlxsw_sp *mlxsw_sp = mlxsw_core_driver_priv(mlxsw_core);
+	struct devlink *devlink = priv_to_devlink(mlxsw_core);
 	int err;
 
 	mlxsw_sp->core = mlxsw_core;
@@ -2978,7 +2986,9 @@ static int mlxsw_sp_init(struct mlxsw_core *mlxsw_core,
 		goto err_sample_trigger_init;
 	}
 
+	devl_lock(devlink);
 	err = mlxsw_sp_ports_create(mlxsw_sp);
+	devl_unlock(devlink);
 	if (err) {
 		dev_err(mlxsw_sp->bus_info->dev, "Failed to create ports\n");
 		goto err_ports_create;
@@ -3159,8 +3169,12 @@ static int mlxsw_sp4_init(struct mlxsw_core *mlxsw_core,
 static void mlxsw_sp_fini(struct mlxsw_core *mlxsw_core)
 {
 	struct mlxsw_sp *mlxsw_sp = mlxsw_core_driver_priv(mlxsw_core);
+	struct devlink *devlink = priv_to_devlink(mlxsw_core);
 
+	devl_lock(devlink);
 	mlxsw_sp_ports_remove(mlxsw_sp);
+	devl_unlock(devlink);
+
 	rhashtable_destroy(&mlxsw_sp->sample_trigger_ht);
 	mlxsw_sp_port_module_info_fini(mlxsw_sp);
 	mlxsw_sp_dpipe_fini(mlxsw_sp);
Signature · 1 line
2.34.1
maintainer
#18 lore raw

Re: [RFT net-next 4/6] eth: mlxsw: switch to explicit locking for port registration

Quoted 7 lines · Thu, Mar 10, 2022 at 01:16:30AM CET, [email protected] wrote:

Explicitly lock the devlink instance and use devl_ API.

This will be used by the subsequent patch to invoke .port_split / .port_unsplit callbacks with devlink instance lock held.

Looks fine. I was about to propose a helpers for the lock/unlock that would take mlxsw_core, but I see you are removing most of them in the next patch :)

author maintainer
#19 lore raw

Re: [RFT net-next 4/6] eth: mlxsw: switch to explicit locking for port registration

Quoted 7 lines · On Thu, 10 Mar 2022 10:17:42 +0100 Jiri Pirko wrote:
Quoted 6 lines · Thu, Mar 10, 2022 at 01:16:30AM CET, [email protected] wrote:

Explicitly lock the devlink instance and use devl_ API.

This will be used by the subsequent patch to invoke .port_split / .port_unsplit callbacks with devlink instance lock held.

Quoted 5 lines
Quoted 1 line

Looks fine. I was about to propose a helpers for the lock/unlock that would take mlxsw_core, but I see you are removing most of them in the next patch :)

Yup :) the end result should be one lock / unlock in probe and remove paths.

author maintainer
#20 lore raw

PATCH 5/6 devlink: hold the instance lock in port_split / port_unsplit callbacks

In reply to Jakub Kicinski

Let the core take the devlink instance lock around port splitting and remove the now redundant locking in the drivers.

---
 .../net/ethernet/mellanox/mlxsw/spectrum.c    |  7 ----
 .../net/ethernet/netronome/nfp/nfp_devlink.c  | 32 +++++--------------
 net/core/devlink.c                            |  2 --
 3 files changed, 8 insertions(+), 33 deletions(-)
drivers/net/ethernet/mellanox/mlxsw/spectrum.c +0 -7
index 1e823b669d1c..8eb05090ffec 100644
@@ -2025,7 +2025,6 @@ static int mlxsw_sp_port_split(struct mlxsw_core *mlxsw_core, u16 local_port,
 			       struct netlink_ext_ack *extack)
 {
 	struct mlxsw_sp *mlxsw_sp = mlxsw_core_driver_priv(mlxsw_core);
-	struct devlink *devlink = priv_to_devlink(mlxsw_core);
 	struct mlxsw_sp_port_mapping port_mapping;
 	struct mlxsw_sp_port *mlxsw_sp_port;
 	enum mlxsw_reg_pmtdb_status status;
@@ -2063,7 +2062,6 @@ static int mlxsw_sp_port_split(struct mlxsw_core *mlxsw_core, u16 local_port,
 
 	port_mapping = mlxsw_sp_port->mapping;
 
-	devl_lock(devlink);
 	for (i = 0; i < count; i++) {
 		u16 s_local_port = mlxsw_reg_pmtdb_port_num_get(pmtdb_pl, i);
 
@@ -2077,13 +2075,11 @@ static int mlxsw_sp_port_split(struct mlxsw_core *mlxsw_core, u16 local_port,
 		dev_err(mlxsw_sp->bus_info->dev, "Failed to create split ports\n");
 		goto err_port_split_create;
 	}
-	devl_unlock(devlink);
 
 	return 0;
 
 err_port_split_create:
 	mlxsw_sp_port_unsplit_create(mlxsw_sp, count, pmtdb_pl);
-	devl_unlock(devlink);
 	return err;
 }
 
@@ -2091,7 +2087,6 @@ static int mlxsw_sp_port_unsplit(struct mlxsw_core *mlxsw_core, u16 local_port,
 				 struct netlink_ext_ack *extack)
 {
 	struct mlxsw_sp *mlxsw_sp = mlxsw_core_driver_priv(mlxsw_core);
-	struct devlink *devlink = priv_to_devlink(mlxsw_core);
 	struct mlxsw_sp_port *mlxsw_sp_port;
 	char pmtdb_pl[MLXSW_REG_PMTDB_LEN];
 	unsigned int count;
@@ -2123,7 +2118,6 @@ static int mlxsw_sp_port_unsplit(struct mlxsw_core *mlxsw_core, u16 local_port,
 		return err;
 	}
 
-	devl_lock(devlink);
 	for (i = 0; i < count; i++) {
 		u16 s_local_port = mlxsw_reg_pmtdb_port_num_get(pmtdb_pl, i);
 
@@ -2132,7 +2126,6 @@ static int mlxsw_sp_port_unsplit(struct mlxsw_core *mlxsw_core, u16 local_port,
 	}
 
 	mlxsw_sp_port_unsplit_create(mlxsw_sp, count, pmtdb_pl);
-	devl_unlock(devlink);
 
 	return 0;
 }
drivers/net/ethernet/netronome/nfp/nfp_devlink.c +8 -24
index 865f62958a72..6bd6f4a67c30 100644
@@ -70,29 +70,21 @@ nfp_devlink_port_split(struct devlink *devlink, unsigned int port_index,
 	unsigned int lanes;
 	int ret;
 
-	devl_lock(devlink);
-
 	rtnl_lock();
 	ret = nfp_devlink_fill_eth_port_from_id(pf, port_index, &eth_port);
 	rtnl_unlock();
 	if (ret)
-		goto out;
+		return ret;
 
-	if (eth_port.port_lanes % count) {
-		ret = -EINVAL;
-		goto out;
-	}
+	if (eth_port.port_lanes % count)
+		return -EINVAL;
 
 	/* Special case the 100G CXP -> 2x40G split */
 	lanes = eth_port.port_lanes / count;
 	if (eth_port.lanes == 10 && count == 2)
 		lanes = 8 / count;
 
-	ret = nfp_devlink_set_lanes(pf, eth_port.index, lanes);
-out:
-	devl_unlock(devlink);
-
-	return ret;
+	return nfp_devlink_set_lanes(pf, eth_port.index, lanes);
 }
 
 static int
@@ -104,29 +96,21 @@ nfp_devlink_port_unsplit(struct devlink *devlink, unsigned int port_index,
 	unsigned int lanes;
 	int ret;
 
-	devl_lock(devlink);
-
 	rtnl_lock();
 	ret = nfp_devlink_fill_eth_port_from_id(pf, port_index, &eth_port);
 	rtnl_unlock();
 	if (ret)
-		goto out;
+		return ret;
 
-	if (!eth_port.is_split) {
-		ret = -EINVAL;
-		goto out;
-	}
+	if (!eth_port.is_split)
+		return -EINVAL;
 
 	/* Special case the 100G CXP -> 2x40G unsplit */
 	lanes = eth_port.port_lanes;
 	if (eth_port.port_lanes == 8)
 		lanes = 10;
 
-	ret = nfp_devlink_set_lanes(pf, eth_port.index, lanes);
-out:
-	devl_unlock(devlink);
-
-	return ret;
+	return nfp_devlink_set_lanes(pf, eth_port.index, lanes);
 }
 
 static int
net/core/devlink.c +0 -2
index c30da1fc023d..3069a3833576 100644
@@ -8676,14 +8676,12 @@ static const struct genl_small_ops devlink_nl_ops[] = {
 		.validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP,
 		.doit = devlink_nl_cmd_port_split_doit,
 		.flags = GENL_ADMIN_PERM,
-		.internal_flags = DEVLINK_NL_FLAG_NO_LOCK,
 	},
 	{
 		.cmd = DEVLINK_CMD_PORT_UNSPLIT,
 		.validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP,
 		.doit = devlink_nl_cmd_port_unsplit_doit,
 		.flags = GENL_ADMIN_PERM,
-		.internal_flags = DEVLINK_NL_FLAG_NO_LOCK,
 	},
 	{
 		.cmd = DEVLINK_CMD_PORT_NEW,
Signature · 1 line
2.34.1
author maintainer
#21 lore raw

PATCH 6/6 devlink: pass devlink_port to port_split / port_unsplit callbacks

In reply to Jakub Kicinski

Now that devlink ports are protected by the instance lock it seems natural to pass devlink_port as an argument to the port_split / port_unsplit callbacks.

This should save the drivers from doing a lookup.

In theory drivers may have supported unsplitting ports which were not registered prior to this change.

---
 drivers/net/ethernet/mellanox/mlxsw/core.c    | 32 ++++++-------
 .../net/ethernet/netronome/nfp/nfp_devlink.c  | 15 +++---
 drivers/net/ethernet/netronome/nfp/nfp_port.c | 18 -------
 drivers/net/ethernet/netronome/nfp/nfp_port.h |  2 -
 include/net/devlink.h                         |  4 +-
 net/core/devlink.c                            | 47 +++++--------------
 6 files changed, 35 insertions(+), 83 deletions(-)
drivers/net/ethernet/mellanox/mlxsw/core.c +14 -18
index e2a6a759eb6c..b13e0f8d232a 100644
@@ -1217,36 +1217,37 @@ static void mlxsw_core_fw_params_unregister(struct mlxsw_core *mlxsw_core)
 				  ARRAY_SIZE(mlxsw_core_fw_devlink_params));
 }
 
+static void *__dl_port(struct devlink_port *devlink_port)
+{
+	return container_of(devlink_port, struct mlxsw_core_port, devlink_port);
+}
+
 static int mlxsw_devlink_port_split(struct devlink *devlink,
-				    unsigned int port_index,
+				    struct devlink_port *port,
 				    unsigned int count,
 				    struct netlink_ext_ack *extack)
 {
+	struct mlxsw_core_port *mlxsw_core_port = __dl_port(port);
 	struct mlxsw_core *mlxsw_core = devlink_priv(devlink);
 
-	if (port_index >= mlxsw_core->max_ports) {
-		NL_SET_ERR_MSG_MOD(extack, "Port index exceeds maximum number of ports");
-		return -EINVAL;
-	}
 	if (!mlxsw_core->driver->port_split)
 		return -EOPNOTSUPP;
-	return mlxsw_core->driver->port_split(mlxsw_core, port_index, count,
-					      extack);
+	return mlxsw_core->driver->port_split(mlxsw_core,
+					      mlxsw_core_port->local_port,
+					      count, extack);
 }
 
 static int mlxsw_devlink_port_unsplit(struct devlink *devlink,
-				      unsigned int port_index,
+				      struct devlink_port *port,
 				      struct netlink_ext_ack *extack)
 {
+	struct mlxsw_core_port *mlxsw_core_port = __dl_port(port);
 	struct mlxsw_core *mlxsw_core = devlink_priv(devlink);
 
-	if (port_index >= mlxsw_core->max_ports) {
-		NL_SET_ERR_MSG_MOD(extack, "Port index exceeds maximum number of ports");
-		return -EINVAL;
-	}
 	if (!mlxsw_core->driver->port_unsplit)
 		return -EOPNOTSUPP;
-	return mlxsw_core->driver->port_unsplit(mlxsw_core, port_index,
+	return mlxsw_core->driver->port_unsplit(mlxsw_core,
+						mlxsw_core_port->local_port,
 						extack);
 }
 
@@ -1280,11 +1281,6 @@ mlxsw_devlink_sb_pool_set(struct devlink *devlink,
 					 extack);
 }
 
-static void *__dl_port(struct devlink_port *devlink_port)
-{
-	return container_of(devlink_port, struct mlxsw_core_port, devlink_port);
-}
-
 static int mlxsw_devlink_port_type_set(struct devlink_port *devlink_port,
 				       enum devlink_port_type port_type)
 {
drivers/net/ethernet/netronome/nfp/nfp_devlink.c +7 -8
index 6bd6f4a67c30..48b95566b52b 100644
@@ -26,12 +26,11 @@ nfp_devlink_fill_eth_port(struct nfp_port *port,
 }
 
 static int
-nfp_devlink_fill_eth_port_from_id(struct nfp_pf *pf, unsigned int port_index,
+nfp_devlink_fill_eth_port_from_id(struct nfp_pf *pf,
+				  struct devlink_port *dl_port,
 				  struct nfp_eth_table_port *copy)
 {
-	struct nfp_port *port;
-
-	port = nfp_port_from_id(pf, NFP_PORT_PHYS_PORT, port_index);
+	struct nfp_port *port = container_of(dl_port, struct nfp_port, dl_port);
 
 	return nfp_devlink_fill_eth_port(port, copy);
 }
@@ -62,7 +61,7 @@ nfp_devlink_set_lanes(struct nfp_pf *pf, unsigned int idx, unsigned int lanes)
 }
 
 static int
-nfp_devlink_port_split(struct devlink *devlink, unsigned int port_index,
+nfp_devlink_port_split(struct devlink *devlink, struct devlink_port *port,
 		       unsigned int count, struct netlink_ext_ack *extack)
 {
 	struct nfp_pf *pf = devlink_priv(devlink);
@@ -71,7 +70,7 @@ nfp_devlink_port_split(struct devlink *devlink, unsigned int port_index,
 	int ret;
 
 	rtnl_lock();
-	ret = nfp_devlink_fill_eth_port_from_id(pf, port_index, &eth_port);
+	ret = nfp_devlink_fill_eth_port_from_id(pf, port, &eth_port);
 	rtnl_unlock();
 	if (ret)
 		return ret;
@@ -88,7 +87,7 @@ nfp_devlink_port_split(struct devlink *devlink, unsigned int port_index,
 }
 
 static int
-nfp_devlink_port_unsplit(struct devlink *devlink, unsigned int port_index,
+nfp_devlink_port_unsplit(struct devlink *devlink, struct devlink_port *port,
 			 struct netlink_ext_ack *extack)
 {
 	struct nfp_pf *pf = devlink_priv(devlink);
@@ -97,7 +96,7 @@ nfp_devlink_port_unsplit(struct devlink *devlink, unsigned int port_index,
 	int ret;
 
 	rtnl_lock();
-	ret = nfp_devlink_fill_eth_port_from_id(pf, port_index, &eth_port);
+	ret = nfp_devlink_fill_eth_port_from_id(pf, port, &eth_port);
 	rtnl_unlock();
 	if (ret)
 		return ret;
drivers/net/ethernet/netronome/nfp/nfp_port.c +0 -18
index 6f3ea1700a15..4f2308570dcf 100644
@@ -75,24 +75,6 @@ int nfp_port_set_features(struct net_device *netdev, netdev_features_t features)
 	return 0;
 }
 
-struct nfp_port *
-nfp_port_from_id(struct nfp_pf *pf, enum nfp_port_type type, unsigned int id)
-{
-	struct devlink *devlink = priv_to_devlink(pf);
-	struct nfp_port *port;
-
-	assert_devl_locked(devlink);
-
-	if (type != NFP_PORT_PHYS_PORT)
-		return NULL;
-
-	list_for_each_entry(port, &pf->ports, port_list)
-		if (port->eth_id == id)
-			return port;
-
-	return NULL;
-}
-
 struct nfp_eth_table_port *__nfp_port_get_eth_port(struct nfp_port *port)
 {
 	if (!port)
drivers/net/ethernet/netronome/nfp/nfp_port.h +0 -2
index df316b9e891d..d1ebe6c72f7f 100644
@@ -106,8 +106,6 @@ nfp_port_set_features(struct net_device *netdev, netdev_features_t features);
 struct nfp_port *nfp_port_from_netdev(struct net_device *netdev);
 int nfp_port_get_port_parent_id(struct net_device *netdev,
 				struct netdev_phys_item_id *ppid);
-struct nfp_port *
-nfp_port_from_id(struct nfp_pf *pf, enum nfp_port_type type, unsigned int id);
 struct nfp_eth_table_port *__nfp_port_get_eth_port(struct nfp_port *port);
 struct nfp_eth_table_port *nfp_port_get_eth_port(struct nfp_port *port);
 
include/net/devlink.h +2 -2
index 9de0d091aee9..fd89a17adea1 100644
@@ -1197,9 +1197,9 @@ struct devlink_ops {
 			 struct netlink_ext_ack *extack);
 	int (*port_type_set)(struct devlink_port *devlink_port,
 			     enum devlink_port_type port_type);
-	int (*port_split)(struct devlink *devlink, unsigned int port_index,
+	int (*port_split)(struct devlink *devlink, struct devlink_port *port,
 			  unsigned int count, struct netlink_ext_ack *extack);
-	int (*port_unsplit)(struct devlink *devlink, unsigned int port_index,
+	int (*port_unsplit)(struct devlink *devlink, struct devlink_port *port,
 			    struct netlink_ext_ack *extack);
 	int (*sb_pool_get)(struct devlink *devlink, unsigned int sb_index,
 			   u16 pool_index,
net/core/devlink.c +12 -35

88 lines not on this page · raw message

Signature · 1 line
2.34.1
#22 lore raw
In reply to Jakub Kicinski
Quoted 10 lines · Jakub Kicinski

This series puts the devlink ports fully under the devlink instance lock's protection. As discussed in the past it implements my preferred solution of exposing the instance lock to the drivers. This way drivers which want to support port splitting can lock the devlink instance themselves on the probe path, and we can take that lock in the core on the split/unsplit paths.

nfp and mlxsw are converted, with slightly deeper changes done in nfp since I'm more familiar with that driver.

Quoted 6 lines

Now that the devlink port is protected we can pass a pointer to the drivers, instead of passing a port index and forcing the drivers to do their own lookups. Both nfp and mlxsw can container_of() to their own structures.

I'd appreciate some testing, I don't have access to this HW.

Thanks for working on this. I ran a few tests that exercise these code paths with a debug config and did not see any immediate problems. I will go over the patches later today

#23 lore raw
Quoted 16 lines · Ido Schimmel
Quoted 15 lines · Jakub Kicinski

This series puts the devlink ports fully under the devlink instance lock's protection. As discussed in the past it implements my preferred solution of exposing the instance lock to the drivers. This way drivers which want to support port splitting can lock the devlink instance themselves on the probe path, and we can take that lock in the core on the split/unsplit paths.

nfp and mlxsw are converted, with slightly deeper changes done in nfp since I'm more familiar with that driver.

Now that the devlink port is protected we can pass a pointer to the drivers, instead of passing a port index and forcing the drivers to do their own lookups. Both nfp and mlxsw can container_of() to their own structures.

Quoted 5 lines
Quoted 1 line

I'd appreciate some testing, I don't have access to this HW.

Thanks for working on this. I ran a few tests that exercise these code paths with a debug config and did not see any immediate problems. I will go over the patches later today

Went over the patches and they look good to me. Thanks again. Will run a full regression with them on Sunday.

I read [1] and [2] again to refresh my memory about this conversion. Can you provide a rough outline of how you plan to go about it? Asking so that I will know what to expect and how it all fits together. I expect that eventually 'DEVLINK_NL_FLAG_NO_LOCK' will be removed from 'DEVLINK_CMD_RELOAD' and then the devl_lock()/devl_unlock() that you left in drivers will be moved to earlier in the probe path so that we don't deadlock on reload.

[1] https://lore.kernel.org/lkml/YYgJ1bnECwUWvNqD@shredder/
[2] https://lore.kernel.org/netdev/[email protected]/

author maintainer
#24 lore raw
Quoted 7 lines · On Thu, 10 Mar 2022 23:13:02 +0200 Ido Schimmel wrote:
Quoted 3 lines · Ido Schimmel

Thanks for working on this. I ran a few tests that exercise these code paths with a debug config and did not see any immediate problems. I will go over the patches later today

Went over the patches and they look good to me. Thanks again. Will run a full regression with them on Sunday.

Thanks!

Quoted 2 lines

I read [1] and [2] again to refresh my memory about this conversion. Can you provide a rough outline of how you plan to go about it?

TBH I haven't started on breaking out more patches, yet. My rough idea was to try to tackle the eswitch callback next and then the reset callback.

Quoted 5 lines

Asking so that I will know what to expect and how it all fits together. I expect that eventually 'DEVLINK_NL_FLAG_NO_LOCK' will be removed from 'DEVLINK_CMD_RELOAD' and then the devl_lock()/devl_unlock() that you left in drivers will be moved to earlier in the probe path so that we don't deadlock on reload.

Yup, that's the end goal 🤞

author maintainer
#25 lore raw
In reply to Ido Schimmel
Quoted 2 lines · On Thu, 10 Mar 2022 23:13:02 +0200 Ido Schimmel wrote:

Went over the patches and they look good to me. Thanks again. Will run a full regression with them on Sunday.

Hi Ido, no news?

Do you have a preference for these patches getting merged for 5.18 or waiting after the merge window? IOW I'm wondering if it's more beneficial for potential backports / out-of-tree builds to have the ability to lock the devlink instance in 5.18 already or to do as much of the conversions as possible in a single release (that'd mean waiting for 5.19)?

If there's no clear preference I'll go for 5.18.

I have the eswitch mode conversion patches almost ready with the "almost" being mlx5. I may just give up on that one and unlock/lock in the handler :S

#26 lore raw
Quoted 5 lines · Jakub Kicinski
Quoted 2 lines · On Thu, 10 Mar 2022 23:13:02 +0200 Ido Schimmel wrote:

Went over the patches and they look good to me. Thanks again. Will run a full regression with them on Sunday.

Hi Ido, no news?

Sorry, forgot to update you. All the tests passed :)

Quoted 9 lines

Do you have a preference for these patches getting merged for 5.18 or waiting after the merge window? IOW I'm wondering if it's more beneficial for potential backports / out-of-tree builds to have the ability to lock the devlink instance in 5.18 already or to do as much of the conversions as possible in a single release (that'd mean waiting for 5.19)?

If there's no clear preference I'll go for 5.18.

5.18 is fine by me.

Thanks!

author maintainer
#27 lore raw
Quoted 3 lines · On Mon, 14 Mar 2022 21:10:31 +0200 Ido Schimmel wrote:
Quoted 1 line

Hi Ido, no news?

Sorry, forgot to update you. All the tests passed :)

Perfect, thank you!

#28 lore raw
In reply to Jakub Kicinski
Quoted 1 line · Jakub Kicinski

On Thu, 10 Mar 2022 23:13:02 +0200 Ido Schimmel wrote:

<...>

Quoted 2 lines

I have the eswitch mode conversion patches almost ready with the "almost" being mlx5.

I wonder why do you need to change eswitch locking in mlx5?

Thanks

author maintainer
#29 lore raw
Quoted 4 lines · On Tue, 15 Mar 2022 09:39:31 +0200 Leon Romanovsky wrote:
Quoted 2 lines

I have the eswitch mode conversion patches almost ready with the "almost" being mlx5.

I wonder why do you need to change eswitch locking in mlx5?

I want DEVLINK_CMD_ESWITCH_SET to drop the DEVLINK_NL_FLAG_NO_LOCK marking.

Other drivers are rather simple in terms of locking (bnxt, nfp, netdevsim) and I can replace driver locking completely with a few minor changes. Other drivers have no locking (insert cry/laugh emoji).

mlx5 has layers and multiple locks, if you're okay with devl_unlock() / devl_lock() inside the callback that's perfect for me.

#30 lore raw
Quoted 8 lines · Jakub Kicinski
Quoted 4 lines · On Tue, 15 Mar 2022 09:39:31 +0200 Leon Romanovsky wrote:
Quoted 2 lines

I have the eswitch mode conversion patches almost ready with the "almost" being mlx5.

I wonder why do you need to change eswitch locking in mlx5?

I want DEVLINK_CMD_ESWITCH_SET to drop the DEVLINK_NL_FLAG_NO_LOCK marking.

+1

Quoted 4 lines

Other drivers are rather simple in terms of locking (bnxt, nfp, netdevsim) and I can replace driver locking completely with a few minor changes. Other drivers have no locking (insert cry/laugh emoji).

I saw it too :)

Quoted 2 lines

mlx5 has layers and multiple locks,

I would say that mlx5 has too many locks.

Quoted 1 line

if you're okay with devl_unlock() / devl_lock() inside the callback that's perfect for me.

The need of DEVLINK_NL_FLAG_NO_LOCK for eswitch is because of questionable locking in devlink_rate_*() calls.

If you success to remove mutex_lock from devlink_rate_nodes_check() and devlink_rate_nodes_destroy(), you won't need devl_unlock/devl_lock for eswitch.

Right now, the eswitch set flow doesn't suffer from races and/or other bugs, just because of global devlink_mutex that protects unlocked parts of devlink_nl_cmd_eswitch_set_doit().

Thanks

maintainer
#31 lore raw
In reply to Jakub Kicinski
Quoted 10 lines · Thu, Mar 10, 2022 at 01:16:26AM CET, [email protected] wrote:

This series puts the devlink ports fully under the devlink instance lock's protection. As discussed in the past it implements my preferred solution of exposing the instance lock to the drivers. This way drivers which want to support port splitting can lock the devlink instance themselves on the probe path, and we can take that lock in the core on the split/unsplit paths.

nfp and mlxsw are converted, with slightly deeper changes done in nfp since I'm more familiar with that driver.

Quoted 6 lines

Now that the devlink port is protected we can pass a pointer to the drivers, instead of passing a port index and forcing the drivers to do their own lookups. Both nfp and mlxsw can container_of() to their own structures.

I'd appreciate some testing, I don't have access to this HW.

I like this patchset. To be honest, I made a mistake to not do have it like this from the beginning. I tried to hide the devlink lock from the user/driver, but as it turned out, that was probably not a good idea :)

Quoted 29 lines

Jakub Kicinski (6):
devlink: expose instance locking and add locked port registering
eth: nfp: wrap locking assertions in helpers
eth: nfp: replace driver's "pf" lock with devlink instance lock
eth: mlxsw: switch to explicit locking for port registration
devlink: hold the instance lock in port_split / port_unsplit callbacks
devlink: pass devlink_port to port_split / port_unsplit callbacks

drivers/net/ethernet/mellanox/mlxsw/core.c    |  36 ++---
drivers/net/ethernet/mellanox/mlxsw/minimal.c |   6 +
.../net/ethernet/mellanox/mlxsw/spectrum.c    |   7 +
.../net/ethernet/netronome/nfp/flower/main.c  |   4 +-
drivers/net/ethernet/netronome/nfp/nfp_app.c  |   2 +-
drivers/net/ethernet/netronome/nfp/nfp_app.h  |  12 +-
.../net/ethernet/netronome/nfp/nfp_devlink.c  |  55 +++----
drivers/net/ethernet/netronome/nfp/nfp_main.c |  19 +--
drivers/net/ethernet/netronome/nfp/nfp_main.h |   6 +-
.../net/ethernet/netronome/nfp/nfp_net_main.c |  34 ++--
.../net/ethernet/netronome/nfp/nfp_net_repr.c |   4 +-
drivers/net/ethernet/netronome/nfp/nfp_port.c |  17 --
drivers/net/ethernet/netronome/nfp/nfp_port.h |   2 -
include/net/devlink.h                         |  15 +-
net/core/devlink.c                            | 148 ++++++++++--------
15 files changed, 196 insertions(+), 171 deletions(-)
Signature · 1 line
2.34.1
#32 lore raw
In reply to Jakub Kicinski
Quoted 17 lines · Jakub Kicinski

This series puts the devlink ports fully under the devlink instance lock's protection. As discussed in the past it implements my preferred solution of exposing the instance lock to the drivers. This way drivers which want to support port splitting can lock the devlink instance themselves on the probe path, and we can take that lock in the core on the split/unsplit paths.

nfp and mlxsw are converted, with slightly deeper changes done in nfp since I'm more familiar with that driver.

Now that the devlink port is protected we can pass a pointer to the drivers, instead of passing a port index and forcing the drivers to do their own lookups. Both nfp and mlxsw can container_of() to their own structures.

I'd appreciate some testing, I don't have access to this HW.

Quoted 7 lines

Jakub Kicinski (6):
devlink: expose instance locking and add locked port registering
eth: nfp: wrap locking assertions in helpers
eth: nfp: replace driver's "pf" lock with devlink instance lock
eth: mlxsw: switch to explicit locking for port registration
devlink: hold the instance lock in port_split / port_unsplit callbacks
devlink: pass devlink_port to port_split / port_unsplit callbacks

Jakub,

Thanks for pursuing in cleanup this devlink mess.

Do you plan to send a series that removes devlink_mutex?

Thanks

Quoted 21 lines
 drivers/net/ethernet/mellanox/mlxsw/core.c    |  36 ++---
 drivers/net/ethernet/mellanox/mlxsw/minimal.c |   6 +
 .../net/ethernet/mellanox/mlxsw/spectrum.c    |   7 +
 .../net/ethernet/netronome/nfp/flower/main.c  |   4 +-
 drivers/net/ethernet/netronome/nfp/nfp_app.c  |   2 +-
 drivers/net/ethernet/netronome/nfp/nfp_app.h  |  12 +-
 .../net/ethernet/netronome/nfp/nfp_devlink.c  |  55 +++----
 drivers/net/ethernet/netronome/nfp/nfp_main.c |  19 +--
 drivers/net/ethernet/netronome/nfp/nfp_main.h |   6 +-
 .../net/ethernet/netronome/nfp/nfp_net_main.c |  34 ++--
 .../net/ethernet/netronome/nfp/nfp_net_repr.c |   4 +-
 drivers/net/ethernet/netronome/nfp/nfp_port.c |  17 --
 drivers/net/ethernet/netronome/nfp/nfp_port.h |   2 -
 include/net/devlink.h                         |  15 +-
 net/core/devlink.c                            | 148 ++++++++++--------
 15 files changed, 196 insertions(+), 171 deletions(-)
Signature · 1 line
2.34.1
author maintainer
#33 lore raw
Quoted 16 lines · On Thu, 10 Mar 2022 11:07:14 +0200 Leon Romanovsky wrote:
Quoted 15 lines · Jakub Kicinski

This series puts the devlink ports fully under the devlink instance lock's protection. As discussed in the past it implements my preferred solution of exposing the instance lock to the drivers. This way drivers which want to support port splitting can lock the devlink instance themselves on the probe path, and we can take that lock in the core on the split/unsplit paths.

nfp and mlxsw are converted, with slightly deeper changes done in nfp since I'm more familiar with that driver.

Now that the devlink port is protected we can pass a pointer to the drivers, instead of passing a port index and forcing the drivers to do their own lookups. Both nfp and mlxsw can container_of() to their own structures.

Quoted 5 lines
Quoted 1 line

I'd appreciate some testing, I don't have access to this HW.

Thanks for pursuing in cleanup this devlink mess.

Do you plan to send a series that removes devlink_mutex?

I would like to convert enough to explicit locking to allow simpler reload handling. I'm happy to leave devlink_mutex removal to someone else, but if there are no takers will do it as well. Let's see how it goes.

#34 lore raw
Quoted 19 lines · Jakub Kicinski
Quoted 18 lines · On Thu, 10 Mar 2022 11:07:14 +0200 Leon Romanovsky wrote:
Quoted 16 lines · Jakub Kicinski

This series puts the devlink ports fully under the devlink instance lock's protection. As discussed in the past it implements my preferred solution of exposing the instance lock to the drivers. This way drivers which want to support port splitting can lock the devlink instance themselves on the probe path, and we can take that lock in the core on the split/unsplit paths.

nfp and mlxsw are converted, with slightly deeper changes done in nfp since I'm more familiar with that driver.

Now that the devlink port is protected we can pass a pointer to the drivers, instead of passing a port index and forcing the drivers to do their own lookups. Both nfp and mlxsw can container_of() to their own structures.

I'd appreciate some testing, I don't have access to this HW.

Quoted 8 lines
Quoted 3 lines

Thanks for pursuing in cleanup this devlink mess.

Do you plan to send a series that removes devlink_mutex?

I would like to convert enough to explicit locking to allow simpler reload handling. I'm happy to leave devlink_mutex removal to someone else, but if there are no takers will do it as well. Let's see how it goes.

Alright, let's see.

The main obstacle to remove devlink_mutex was netdevsim port creation sysfs, so after your locking series, that change will be more or less trivial.

Thanks

#35 lore raw
In reply to Jakub Kicinski
Quoted 10 lines · Jakub Kicinski

This series puts the devlink ports fully under the devlink instance lock's protection. As discussed in the past it implements my preferred solution of exposing the instance lock to the drivers. This way drivers which want to support port splitting can lock the devlink instance themselves on the probe path, and we can take that lock in the core on the split/unsplit paths.

nfp and mlxsw are converted, with slightly deeper changes done in nfp since I'm more familiar with that driver.

Quoted 6 lines

Now that the devlink port is protected we can pass a pointer to the drivers, instead of passing a port index and forcing the drivers to do their own lookups. Both nfp and mlxsw can container_of() to their own structures.

I'd appreciate some testing, I don't have access to this HW.

Thanks Jakub,

this looks like a nice improvement :)

We have exercised this patch-set on NFP-based NICs and have not observed any regressions.

author maintainer
#36 lore raw
Quoted 6 lines · On Fri, 11 Mar 2022 11:48:53 +0100 Simon Horman wrote:

Thanks Jakub,

this looks like a nice improvement :)

We have exercised this patch-set on NFP-based NICs and have not observed any regressions.

Awesome, thanks a lot for testing! :)