Skip to contentSkip to conversation

devlink: remove the wait-for-references on unregister

RFC In review Jakub Kicinski 3 participants 50 messages Dec 17, 2022 – Jan 4, 2023

Subsystems (4)
lore
Share & tools
lore

Jakub Kicinski

author maintainer
#0 lore raw

cover devlink: remove the wait-for-references on unregister

This set is on top of the previous RFC.

Move the registration and unregistration of the devlink instances under their instance locks. Don't perform the netdev-style wait for all references when unregistering the instance.

Jakub Kicinski (10):
devlink: bump the instance index directly when iterating
devlink: update the code in netns move to latest helpers
devlink: protect devlink->dev by the instance lock
devlink: always check if the devlink instance is registered
devlink: remove the registration guarantee of references
devlink: don't require setting features before registration
netdevsim: rename a label
netdevsim: move devlink registration under the instance lock
devlink: allow registering parameters after the instance
netdevsim: register devlink instance before sub-objects

 drivers/net/netdevsim/dev.c |  15 +++--
 include/net/devlink.h       |   3 +
 net/devlink/basic.c         |  64 ++++++++++++------
 net/devlink/core.c          | 127 +++++++++++++++++-------------------
 net/devlink/devl_internal.h |  20 ++----
 net/devlink/netlink.c       |  19 ++++--
 6 files changed, 136 insertions(+), 112 deletions(-)
Signature · 1 line
2.38.1
author maintainer
#1 lore raw

RFC 1/10 devlink: bump the instance index directly when iterating

We use a clever find_first() / find_after() scheme currently, which works nicely as xarray will write the "current" index into the variable we pass.

We can't do the same thing during the "dump walk" because there we must not increment the index until we're sure that the instance has been fully dumped.

Since we have a precedent and a requirement for manually futzing with the increment of the index, let's switch devlinks_xa_for_each_registered_get() to do the same thing. It removes some indirections.

---
 net/devlink/core.c          | 31 +++++++++----------------------
 net/devlink/devl_internal.h | 17 ++++-------------
 2 files changed, 13 insertions(+), 35 deletions(-)
net/devlink/core.c +9 -22
index 371d6821315d..88c88b8053e2 100644
@@ -91,16 +91,13 @@ void devlink_put(struct devlink *devlink)
 		call_rcu(&devlink->rcu, __devlink_put_rcu);
 }
 
-struct devlink *
-devlinks_xa_find_get(struct net *net, unsigned long *indexp,
-		     void * (*xa_find_fn)(struct xarray *, unsigned long *,
-					  unsigned long, xa_mark_t))
+struct devlink *devlinks_xa_find_get(struct net *net, unsigned long *indexp)
 {
-	struct devlink *devlink;
+	struct devlink *devlink = NULL;
 
 	rcu_read_lock();
 retry:
-	devlink = xa_find_fn(&devlinks, indexp, ULONG_MAX, DEVLINK_REGISTERED);
+	devlink = xa_find(&devlinks, indexp, ULONG_MAX, DEVLINK_REGISTERED);
 	if (!devlink)
 		goto unlock;
 
@@ -109,31 +106,21 @@ devlinks_xa_find_get(struct net *net, unsigned long *indexp,
 	 * This prevents live-lock of devlink_unregister() wait for completion.
 	 */
 	if (xa_get_mark(&devlinks, *indexp, DEVLINK_UNREGISTERING))
-		goto retry;
+		goto next;
 
-	/* For a possible retry, the xa_find_after() should be always used */
-	xa_find_fn = xa_find_after;
 	if (!devlink_try_get(devlink))
-		goto retry;
+		goto next;
 	if (!net_eq(devlink_net(devlink), net)) {
 		devlink_put(devlink);
-		goto retry;
+		goto next;
 	}
 unlock:
 	rcu_read_unlock();
 	return devlink;
-}
-
-struct devlink *
-devlinks_xa_find_get_first(struct net *net, unsigned long *indexp)
-{
-	return devlinks_xa_find_get(net, indexp, xa_find);
-}
 
-struct devlink *
-devlinks_xa_find_get_next(struct net *net, unsigned long *indexp)
-{
-	return devlinks_xa_find_get(net, indexp, xa_find_after);
+next:
+	(*indexp)++;
+	goto retry;
 }
 
 /**
net/devlink/devl_internal.h +4 -13
index 1d7ab11f2f7e..ef0369449592 100644
@@ -82,18 +82,9 @@ extern struct genl_family devlink_nl_family;
  * in loop body in order to release the reference.
  */
 #define devlinks_xa_for_each_registered_get(net, index, devlink)	\
-	for (index = 0,							\
-	     devlink = devlinks_xa_find_get_first(net, &index);	\
-	     devlink; devlink = devlinks_xa_find_get_next(net, &index))
-
-struct devlink *
-devlinks_xa_find_get(struct net *net, unsigned long *indexp,
-		     void * (*xa_find_fn)(struct xarray *, unsigned long *,
-					  unsigned long, xa_mark_t));
-struct devlink *
-devlinks_xa_find_get_first(struct net *net, unsigned long *indexp);
-struct devlink *
-devlinks_xa_find_get_next(struct net *net, unsigned long *indexp);
+	for (index = 0; (devlink = devlinks_xa_find_get(net, &index)); index++)
+
+struct devlink *devlinks_xa_find_get(struct net *net, unsigned long *indexp);
 
 /* Netlink */
 #define DEVLINK_NL_FLAG_NEED_PORT		BIT(0)
@@ -133,7 +124,7 @@ struct devlink_gen_cmd {
  */
 #define devlink_dump_for_each_instance_get(msg, dump, devlink)		\
 	for (; (devlink = devlinks_xa_find_get(sock_net(msg->sk),	\
-					       &dump->instance, xa_find)); \
+					       &dump->instance));	\
 	     dump->instance++, dump->idx = 0)
 
 extern const struct genl_small_ops devlink_nl_ops[56];
Signature · 1 line
2.38.1
maintainer
#2 lore raw

Re: [RFC net-next 01/10] devlink: bump the instance index directly when iterating

Quoted 7 lines · Sat, Dec 17, 2022 at 02:19:44AM CET, [email protected] wrote:

We use a clever find_first() / find_after() scheme currently, which works nicely as xarray will write the "current" index into the variable we pass.

We can't do the same thing during the "dump walk" because there we must not increment the index until we're sure that the instance has been fully dumped.

To be honest, this "we something" desctiption style makes things quite hard to understand. Could you please rephrase it to actually talk about the entities in code?

Quoted 37 lines

Since we have a precedent and a requirement for manually futzing with the increment of the index, let's switch devlinks_xa_for_each_registered_get() to do the same thing. It removes some indirections.

---
net/devlink/core.c | 31 +++++++++----------------------
net/devlink/devl_internal.h | 17 ++++-------------

2 files changed, 13 insertions(+), 35 deletions(-)
net/devlink/core.c +2 -5
index 371d6821315d..88c88b8053e2 100644
@@ -91,16 +91,13 @@ void devlink_put(struct devlink *devlink)
 		call_rcu(&devlink->rcu, __devlink_put_rcu);
 }
 
-struct devlink *
-devlinks_xa_find_get(struct net *net, unsigned long *indexp,
-		     void * (*xa_find_fn)(struct xarray *, unsigned long *,
-					  unsigned long, xa_mark_t))
+struct devlink *devlinks_xa_find_get(struct net *net, unsigned long *indexp)
 {
-	struct devlink *devlink;
+	struct devlink *devlink = NULL;
 
 	rcu_read_lock();

retry:

net/devlink/core.c +1 -1
-	devlink = xa_find_fn(&devlinks, indexp, ULONG_MAX, DEVLINK_REGISTERED);
+	devlink = xa_find(&devlinks, indexp, ULONG_MAX, DEVLINK_REGISTERED);
	if (!devlink)
		goto unlock;

net/devlink/core.c devlinks_xa_find_get(struct net *net, unsigned long *indexp, · line 115

Quoted 8 lines
net/devlink/core.c +1 -3
@@ -109,31 +106,21 @@ devlinks_xa_find_get(struct net *net, unsigned long *indexp,
 	 * This prevents live-lock of devlink_unregister() wait for completion.
 	 */
 	if (xa_get_mark(&devlinks, *indexp, DEVLINK_UNREGISTERING))
-		goto retry;
+		goto next;
 
-	/* For a possible retry, the xa_find_after() should be always used */
-	xa_find_fn = xa_find_after;

Hmm. Any idea why xa_find_after()? implementation is different to xa_find()?

Quoted 40 lines
net/devlink/core.c +2 -2
 	if (!devlink_try_get(devlink))
-		goto retry;
+		goto next;
 	if (!net_eq(devlink_net(devlink), net)) {
 		devlink_put(devlink);
-		goto retry;
+		goto next;
 	}

unlock:

	rcu_read_unlock();
	return devlink;
net/devlink/core.c +3 -11
-}
-
-struct devlink *
-devlinks_xa_find_get_first(struct net *net, unsigned long *indexp)
-{
-	return devlinks_xa_find_get(net, indexp, xa_find);
-}
 
-struct devlink *
-devlinks_xa_find_get_next(struct net *net, unsigned long *indexp)
-{
-	return devlinks_xa_find_get(net, indexp, xa_find_after);
+next:
+	(*indexp)++;
+	goto retry;

}

/**

net/devlink/devl_internal.h +0 -3
index 1d7ab11f2f7e..ef0369449592 100644
@@ -82,18 +82,9 @@ extern struct genl_family devlink_nl_family;
 * in loop body in order to release the reference.
 */
 #define devlinks_xa_for_each_registered_get(net, index, devlink)	\
-	for (index = 0,							\
-	     devlink = devlinks_xa_find_get_first(net, &index);	\
-	     devlink; devlink = devlinks_xa_find_get_next(net, &index))

net/devlink/devl_internal.h extern struct genl_family devlink_nl_family; · line 85

Quoted 10 lines
net/devlink/devl_internal.h +1 -9
-
-struct devlink *
-devlinks_xa_find_get(struct net *net, unsigned long *indexp,
-		     void * (*xa_find_fn)(struct xarray *, unsigned long *,
-					  unsigned long, xa_mark_t));
-struct devlink *
-devlinks_xa_find_get_first(struct net *net, unsigned long *indexp);
-struct devlink *
-devlinks_xa_find_get_next(struct net *net, unsigned long *indexp);
+	for (index = 0; (devlink = devlinks_xa_find_get(net, &index)); index++)

You don't need ()' in the 2nd for arg.

Quoted 17 lines
net/devlink/devl_internal.h +3 -1
+
+struct devlink *devlinks_xa_find_get(struct net *net, unsigned long *indexp);
 
 /* Netlink */
 #define DEVLINK_NL_FLAG_NEED_PORT		BIT(0)
@@ -133,7 +124,7 @@ struct devlink_gen_cmd {
 */
 #define devlink_dump_for_each_instance_get(msg, dump, devlink)		\
 	for (; (devlink = devlinks_xa_find_get(sock_net(msg->sk),	\
-					       &dump->instance, xa_find)); \
+					       &dump->instance));	\
 	     dump->instance++, dump->idx = 0)
 
 extern const struct genl_small_ops devlink_nl_ops[56];
Signature · 1 line
2.38.1
author maintainer
#3 lore raw

Re: [RFC net-next 01/10] devlink: bump the instance index directly when iterating

Quoted 5 lines · On Mon, 2 Jan 2023 14:24:39 +0100 Jiri Pirko wrote:
Quoted 4 lines · Sat, Dec 17, 2022 at 02:19:44AM CET, [email protected] wrote:

We use a clever find_first() / find_after() scheme currently, which works nicely as xarray will write the "current" index into the variable we pass.

Quoted 7 lines
Quoted 3 lines

We can't do the same thing during the "dump walk" because there we must not increment the index until we're sure that the instance has been fully dumped.

To be honest, this "we something" desctiption style makes things quite hard to understand. Could you please rephrase it to actually talk about the entities in code?

Could you be more specific? I'm probably misunderstanding but it sounds to me like you're asking me to describe what I'm doing rather than the background and motivation.

Quoted 15 lines
Quoted 15 lines
net/devlink/devl_internal.h +0 -7
index 1d7ab11f2f7e..ef0369449592 100644
@@ -82,18 +82,9 @@ extern struct genl_family devlink_nl_family;
 * in loop body in order to release the reference.
 */
 #define devlinks_xa_for_each_registered_get(net, index, devlink)	\
-	for (index = 0,							\
-	     devlink = devlinks_xa_find_get_first(net, &index);	\
-	     devlink; devlink = devlinks_xa_find_get_next(net, &index))
-
-struct devlink *
-devlinks_xa_find_get(struct net *net, unsigned long *indexp,
-		     void * (*xa_find_fn)(struct xarray *, unsigned long *,
Quoted 8 lines

net/devlink/devl_internal.h extern struct genl_family devlink_nl_family; · line 85

Quoted 6 lines
net/devlink/devl_internal.h +1 -5
-					  unsigned long, xa_mark_t));
-struct devlink *
-devlinks_xa_find_get_first(struct net *net, unsigned long *indexp);
-struct devlink *
-devlinks_xa_find_get_next(struct net *net, unsigned long *indexp);
+	for (index = 0; (devlink = devlinks_xa_find_get(net, &index)); index++)  

You don't need ()' in the 2nd for arg.

Pretty sure I was getting a compiler warning for assignment in a condition....

maintainer
#4 lore raw

Re: [RFC net-next 01/10] devlink: bump the instance index directly when iterating

Quoted 10 lines · Mon, Jan 02, 2023 at 11:48:13PM CET, [email protected] wrote:
Quoted 9 lines · On Mon, 2 Jan 2023 14:24:39 +0100 Jiri Pirko wrote:
Quoted 7 lines · Sat, Dec 17, 2022 at 02:19:44AM CET, [email protected] wrote:

We use a clever find_first() / find_after() scheme currently, which works nicely as xarray will write the "current" index into the variable we pass.

We can't do the same thing during the "dump walk" because there we must not increment the index until we're sure that the instance has been fully dumped.

Quoted 7 lines
Quoted 3 lines

To be honest, this "we something" desctiption style makes things quite hard to understand. Could you please rephrase it to actually talk about the entities in code?

Could you be more specific? I'm probably misunderstanding but it sounds to me like you're asking me to describe what I'm doing rather than the background and motivation.

"We" is us, you me and other people. It is weird to talk about the code and what there as "we do", "we use" etc. It's quite confusing as the reader it not able to distinguish if you are talking about code or people (developers in this case). I'm just askin if it is possible to make the descriptions easier to understand.

Quoted 25 lines
Quoted 16 lines
Quoted 16 lines
net/devlink/devl_internal.h +0 -7
index 1d7ab11f2f7e..ef0369449592 100644
@@ -82,18 +82,9 @@ extern struct genl_family devlink_nl_family;
 * in loop body in order to release the reference.
 */
 #define devlinks_xa_for_each_registered_get(net, index, devlink)	\
-	for (index = 0,							\
-	     devlink = devlinks_xa_find_get_first(net, &index);	\
-	     devlink; devlink = devlinks_xa_find_get_next(net, &index))
-
-struct devlink *
-devlinks_xa_find_get(struct net *net, unsigned long *indexp,
-		     void * (*xa_find_fn)(struct xarray *, unsigned long *,
Quoted 8 lines

net/devlink/devl_internal.h extern struct genl_family devlink_nl_family; · line 85

Quoted 6 lines
net/devlink/devl_internal.h +1 -5
-					  unsigned long, xa_mark_t));
-struct devlink *
-devlinks_xa_find_get_first(struct net *net, unsigned long *indexp);
-struct devlink *
-devlinks_xa_find_get_next(struct net *net, unsigned long *indexp);
+	for (index = 0; (devlink = devlinks_xa_find_get(net, &index)); index++)  

You don't need ()' in the 2nd for arg.

Quoted 2 lines

Pretty sure I was getting a compiler warning for assignment in a condition....

I see.

author maintainer
#5 lore raw

Re: [RFC net-next 01/10] devlink: bump the instance index directly when iterating

Quoted 8 lines · On Tue, 3 Jan 2023 08:35:16 +0100 Jiri Pirko wrote:
Quoted 7 lines
Quoted 3 lines

To be honest, this "we something" desctiption style makes things quite hard to understand. Could you please rephrase it to actually talk about the entities in code?

Could you be more specific? I'm probably misunderstanding but it sounds to me like you're asking me to describe what I'm doing rather than the background and motivation.

Quoted 5 lines

"We" is us, you me and other people. It is weird to talk about the code and what there as "we do", "we use" etc. It's quite confusing as the reader it not able to distinguish if you are talking about code or people (developers in this case). I'm just askin if it is possible to make the descriptions easier to understand.

Actually thinking about it again, this patch is not strictly necessary. I'll rewrite the commit message to just say it's a simplification based on the fact we don't have multi-index entries.

author maintainer
#6 lore raw

Re: [RFC net-next 01/10] devlink: bump the instance index directly when iterating

In reply to Jiri Pirko
Quoted 23 lines · On Mon, 2 Jan 2023 14:24:39 +0100 Jiri Pirko wrote:
Quoted 23 lines
net/devlink/core.c +1 -4
-struct devlink *
-devlinks_xa_find_get(struct net *net, unsigned long *indexp,
-		     void * (*xa_find_fn)(struct xarray *, unsigned long *,
-					  unsigned long, xa_mark_t))
+struct devlink *devlinks_xa_find_get(struct net *net, unsigned long *indexp)

{

net/devlink/core.c +1 -1
-	struct devlink *devlink;
+	struct devlink *devlink = NULL;
	rcu_read_lock();

retry:

net/devlink/core.c +1 -1
-	devlink = xa_find_fn(&devlinks, indexp, ULONG_MAX, DEVLINK_REGISTERED);
+	devlink = xa_find(&devlinks, indexp, ULONG_MAX, DEVLINK_REGISTERED);
	if (!devlink)
		goto unlock;
net/devlink/core.c +1 -1
@@ -109,31 +106,21 @@ devlinks_xa_find_get(struct net *net, unsigned long *indexp,
 	 * This prevents live-lock of devlink_unregister() wait for completion.
 	 */
 	if (xa_get_mark(&devlinks, *indexp, DEVLINK_UNREGISTERING))
-		goto retry;
+		goto next;
 
Quoted 5 lines

net/devlink/core.c devlinks_xa_find_get(struct net *net, unsigned long *indexp, · line 115

Quoted 2 lines
net/devlink/core.c +0 -2
-	/* For a possible retry, the xa_find_after() should be always used */
-	xa_find_fn = xa_find_after;  

Hmm. Any idea why xa_find_after()? implementation is different to xa_find()?

I'm guessing it's because for _after the code needs to take special care of skipping multi-index entries. If an entry spans 0-3 xa_find(0) can return the same entry as xa_find(1). But xa_find_after(1) must return an entry under index 4 or higher.

Since our use of the Xarray is very trivial with no range indexes, it should not matter.

Let me CC Matthew, just in case. The question boils down to whether:

	for (index = 0; (entry = xa_find(net, &index));	index++)  

is a legal way of iterating over an Xarray without range indexes.

Quoted 50 lines
Quoted 50 lines
net/devlink/core.c +2 -2
 	if (!devlink_try_get(devlink))
-		goto retry;
+		goto next;
 	if (!net_eq(devlink_net(devlink), net)) {
 		devlink_put(devlink);
-		goto retry;
+		goto next;
 	}

unlock:

	rcu_read_unlock();
	return devlink;
net/devlink/core.c +3 -11
-}
-
-struct devlink *
-devlinks_xa_find_get_first(struct net *net, unsigned long *indexp)
-{
-	return devlinks_xa_find_get(net, indexp, xa_find);
-}
 
-struct devlink *
-devlinks_xa_find_get_next(struct net *net, unsigned long *indexp)
-{
-	return devlinks_xa_find_get(net, indexp, xa_find_after);
+next:
+	(*indexp)++;
+	goto retry;

}

/**

net/devlink/devl_internal.h +1 -12
index 1d7ab11f2f7e..ef0369449592 100644
@@ -82,18 +82,9 @@ extern struct genl_family devlink_nl_family;
 * in loop body in order to release the reference.
 */
 #define devlinks_xa_for_each_registered_get(net, index, devlink)	\
-	for (index = 0,							\
-	     devlink = devlinks_xa_find_get_first(net, &index);	\
-	     devlink; devlink = devlinks_xa_find_get_next(net, &index))
-
-struct devlink *
-devlinks_xa_find_get(struct net *net, unsigned long *indexp,
-		     void * (*xa_find_fn)(struct xarray *, unsigned long *,
-					  unsigned long, xa_mark_t));
-struct devlink *
-devlinks_xa_find_get_first(struct net *net, unsigned long *indexp);
-struct devlink *
-devlinks_xa_find_get_next(struct net *net, unsigned long *indexp);
+	for (index = 0; (devlink = devlinks_xa_find_get(net, &index)); index++)  
author maintainer
#7 lore raw

RFC 2/10 devlink: update the code in netns move to latest helpers

In reply to Jakub Kicinski

devlink_pernet_pre_exit() is the only obvious place which takes the instance lock without using the devl_ helpers. Update the code and move the error print after releasing the reference (having unlock and put together feels slightly idiomatic).

---
 net/devlink/core.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
net/devlink/core.c +4 -3
index 88c88b8053e2..d3b8336946fd 100644
@@ -299,15 +299,16 @@ static void __net_exit devlink_pernet_pre_exit(struct net *net)
 	 */
 	devlinks_xa_for_each_registered_get(net, index, devlink) {
 		WARN_ON(!(devlink->features & DEVLINK_F_RELOAD));
-		mutex_lock(&devlink->lock);
+		devl_lock(devlink);
 		err = devlink_reload(devlink, &init_net,
 				     DEVLINK_RELOAD_ACTION_DRIVER_REINIT,
 				     DEVLINK_RELOAD_LIMIT_UNSPEC,
 				     &actions_performed, NULL);
-		mutex_unlock(&devlink->lock);
+		devl_unlock(devlink);
+		devlink_put(devlink);
+
 		if (err && err != -EOPNOTSUPP)
 			pr_warn("Failed to reload devlink instance into init_net\n");
-		devlink_put(devlink);
 	}
 }
 
Signature · 1 line
2.38.1
maintainer Reviewed-by on v1 2/10
#8 lore raw

Re: [RFC net-next 02/10] devlink: update the code in netns move to latest helpers

Quoted 4 lines · Sat, Dec 17, 2022 at 02:19:45AM CET, [email protected] wrote:

devlink_pernet_pre_exit() is the only obvious place which takes the instance lock without using the devl_ helpers. Update the code and move the error print after releasing the reference (having unlock and put together feels slightly idiomatic).

To be prudent, this should be 2 patches as it changes 2 separate things. Anyway,

author maintainer
#9 lore raw

RFC 3/10 devlink: protect devlink->dev by the instance lock

In reply to Jakub Kicinski

devlink->dev is assumed to be always valid as long as any outstanding reference to the devlink instance exists.

In prep for weakening of the references take the instance lock.

---
 net/devlink/basic.c         | 7 +++----
 net/devlink/devl_internal.h | 3 ++-
 net/devlink/netlink.c       | 9 ++++++---
 3 files changed, 11 insertions(+), 8 deletions(-)
net/devlink/basic.c +3 -4
index ceac4343698b..5f33d74eef83 100644
@@ -6314,12 +6314,10 @@ static int devlink_nl_cmd_region_read_dumpit(struct sk_buff *skb,
 
 	start_offset = dump->start_offset;
 
-	devlink = devlink_get_from_attrs(sock_net(cb->skb->sk), attrs);
+	devlink = devlink_get_from_attrs_lock(sock_net(cb->skb->sk), attrs);
 	if (IS_ERR(devlink))
 		return PTR_ERR(devlink);
 
-	devl_lock(devlink);
-
 	if (!attrs[DEVLINK_ATTR_REGION_NAME]) {
 		NL_SET_ERR_MSG(cb->extack, "No region name provided");
 		err = -EINVAL;
@@ -7735,9 +7733,10 @@ devlink_health_reporter_get_from_cb(struct netlink_callback *cb)
 	struct nlattr **attrs = info->attrs;
 	struct devlink *devlink;
 
-	devlink = devlink_get_from_attrs(sock_net(cb->skb->sk), attrs);
+	devlink = devlink_get_from_attrs_lock(sock_net(cb->skb->sk), attrs);
 	if (IS_ERR(devlink))
 		return NULL;
+	devl_unlock(devlink);
 
 	reporter = devlink_health_reporter_get_from_attrs(devlink, attrs);
 	devlink_put(devlink);
net/devlink/devl_internal.h +2 -1
index ef0369449592..c3977c69552a 100644
@@ -129,7 +129,8 @@ struct devlink_gen_cmd {
 
 extern const struct genl_small_ops devlink_nl_ops[56];
 
-struct devlink *devlink_get_from_attrs(struct net *net, struct nlattr **attrs);
+struct devlink *
+devlink_get_from_attrs_lock(struct net *net, struct nlattr **attrs);
 
 void devlink_notify_unregister(struct devlink *devlink);
 void devlink_notify_register(struct devlink *devlink);
net/devlink/netlink.c +6 -3
index 7eb39ccb2ae7..b38df704be1c 100644
@@ -82,7 +82,8 @@ static const struct nla_policy devlink_nl_policy[DEVLINK_ATTR_MAX + 1] = {
 	[DEVLINK_ATTR_REGION_DIRECT] = { .type = NLA_FLAG },
 };
 
-struct devlink *devlink_get_from_attrs(struct net *net, struct nlattr **attrs)
+struct devlink *
+devlink_get_from_attrs_lock(struct net *net, struct nlattr **attrs)
 {
 	struct devlink *devlink;
 	unsigned long index;
@@ -96,9 +97,11 @@ struct devlink *devlink_get_from_attrs(struct net *net, struct nlattr **attrs)
 	devname = nla_data(attrs[DEVLINK_ATTR_DEV_NAME]);
 
 	devlinks_xa_for_each_registered_get(net, index, devlink) {
+		devl_lock(devlink);
 		if (strcmp(devlink->dev->bus->name, busname) == 0 &&
 		    strcmp(dev_name(devlink->dev), devname) == 0)
 			return devlink;
+		devl_unlock(devlink);
 		devlink_put(devlink);
 	}
 
@@ -113,10 +116,10 @@ static int devlink_nl_pre_doit(const struct genl_split_ops *ops,
 	struct devlink *devlink;
 	int err;
 
-	devlink = devlink_get_from_attrs(genl_info_net(info), info->attrs);
+	devlink = devlink_get_from_attrs_lock(genl_info_net(info), info->attrs);
 	if (IS_ERR(devlink))
 		return PTR_ERR(devlink);
-	devl_lock(devlink);
+
 	info->user_ptr[0] = devlink;
 	if (ops->internal_flags & DEVLINK_NL_FLAG_NEED_PORT) {
 		devlink_port = devlink_port_get_from_info(devlink, info);
Signature · 1 line
2.38.1
author maintainer
#10 lore raw

RFC 4/10 devlink: always check if the devlink instance is registered

In reply to Jakub Kicinski

Always check under the instance lock whether the devlink instance is still / already registered.

This is a no-op for the most part, as the unregistration path currently waits for all references. On the init path, however, we may temporarily open up a race with netdev code, if netdevs are registered before the devlink instance. This is temporary, the next change fixes it, and this commit has been split out for the ease of review.

Note that in case of iterating over sub-objects which have their own lock (regions and line cards) we assume an implicit dependency between those objects existing and devlink unregistration.

---
 include/net/devlink.h |  1 +
 net/devlink/basic.c   | 35 +++++++++++++++++++++++++++++------
 net/devlink/core.c    | 25 +++++++++++++++++++++----
 net/devlink/netlink.c | 10 ++++++++--
 4 files changed, 59 insertions(+), 12 deletions(-)
include/net/devlink.h +1 -0
index 6a2e4f21779f..36e013d3aa52 100644
@@ -1626,6 +1626,7 @@ struct device *devlink_to_dev(const struct devlink *devlink);
 void devl_lock(struct devlink *devlink);
 int devl_trylock(struct devlink *devlink);
 void devl_unlock(struct devlink *devlink);
+bool devl_is_alive(struct devlink *devlink);
 void devl_assert_locked(struct devlink *devlink);
 bool devl_lock_is_held(struct devlink *devlink);
 
net/devlink/basic.c +29 -6
index 5f33d74eef83..6b18e70a39fd 100644
@@ -2130,6 +2130,9 @@ static int devlink_nl_cmd_linecard_get_dumpit(struct sk_buff *msg,
 		int idx = 0;
 
 		mutex_lock(&devlink->linecards_lock);
+		if (!devl_is_alive(devlink))
+			goto next_devlink;
+
 		list_for_each_entry(linecard, &devlink->linecard_list, list) {
 			if (idx < dump->idx) {
 				idx++;
@@ -2151,6 +2154,7 @@ static int devlink_nl_cmd_linecard_get_dumpit(struct sk_buff *msg,
 			}
 			idx++;
 		}
+next_devlink:
 		mutex_unlock(&devlink->linecards_lock);
 		devlink_put(devlink);
 	}
@@ -7809,6 +7813,12 @@ devlink_nl_cmd_health_reporter_get_dumpit(struct sk_buff *msg,
 		int idx = 0;
 
 		mutex_lock(&devlink->reporters_lock);
+		if (!devl_is_alive(devlink)) {
+			mutex_unlock(&devlink->reporters_lock);
+			devlink_put(devlink);
+			continue;
+		}
+
 		list_for_each_entry(reporter, &devlink->reporter_list,
 				    list) {
 			if (idx < dump->idx) {
@@ -7830,6 +7840,9 @@ devlink_nl_cmd_health_reporter_get_dumpit(struct sk_buff *msg,
 		mutex_unlock(&devlink->reporters_lock);
 
 		devl_lock(devlink);
+		if (!devl_is_alive(devlink))
+			goto next_devlink;
+
 		xa_for_each(&devlink->ports, port_index, port) {
 			mutex_lock(&port->reporters_lock);
 			list_for_each_entry(reporter, &port->reporter_list, list) {
@@ -7853,6 +7866,7 @@ devlink_nl_cmd_health_reporter_get_dumpit(struct sk_buff *msg,
 			}
 			mutex_unlock(&port->reporters_lock);
 		}
+next_devlink:
 		devl_unlock(devlink);
 		devlink_put(devlink);
 	}
@@ -12218,7 +12232,8 @@ void devlink_compat_running_version(struct devlink *devlink,
 		return;
 
 	devl_lock(devlink);
-	__devlink_compat_running_version(devlink, buf, len);
+	if (devl_is_alive(devlink))
+		__devlink_compat_running_version(devlink, buf, len);
 	devl_unlock(devlink);
 }
 
@@ -12227,20 +12242,28 @@ int devlink_compat_flash_update(struct devlink *devlink, const char *file_name)
 	struct devlink_flash_update_params params = {};
 	int ret;
 
-	if (!devlink->ops->flash_update)
-		return -EOPNOTSUPP;
+	devl_lock(devlink);
+	if (!devl_is_alive(devlink)) {
+		ret = -ENODEV;
+		goto out_unlock;
+	}
+
+	if (!devlink->ops->flash_update) {
+		ret = -EOPNOTSUPP;
+		goto out_unlock;
+	}
 
 	ret = request_firmware(&params.fw, file_name, devlink->dev);
 	if (ret)
-		return ret;
+		goto out_unlock;
 
-	devl_lock(devlink);
 	devlink_flash_update_begin_notify(devlink);
 	ret = devlink->ops->flash_update(devlink, &params, NULL);
 	devlink_flash_update_end_notify(devlink);
-	devl_unlock(devlink);
 
 	release_firmware(params.fw);
+out_unlock:
+	devl_unlock(devlink);
 
 	return ret;
 }
net/devlink/core.c +21 -4
index d3b8336946fd..2abad8247597 100644
@@ -67,6 +67,21 @@ void devl_unlock(struct devlink *devlink)
 }
 EXPORT_SYMBOL_GPL(devl_unlock);
 
+bool devl_is_alive(struct devlink *devlink)
+{
+	return xa_get_mark(&devlinks, devlink->index, DEVLINK_REGISTERED);
+}
+EXPORT_SYMBOL_GPL(devl_is_alive);
+
+/**
+ * devlink_try_get() - try to obtain a reference on a devlink instance
+ * @devlink: instance to reference
+ *
+ * Obtain a reference on a devlink instance. A reference on a devlink instance
+ * only implies that it's safe to take the instance lock. It does not imply
+ * that the instance is registered, use devl_is_alive() after taking
+ * the instance lock to check registration status.
+ */
 struct devlink *__must_check devlink_try_get(struct devlink *devlink)
 {
 	if (refcount_inc_not_zero(&devlink->refcount))
@@ -300,10 +315,12 @@ static void __net_exit devlink_pernet_pre_exit(struct net *net)
 	devlinks_xa_for_each_registered_get(net, index, devlink) {
 		WARN_ON(!(devlink->features & DEVLINK_F_RELOAD));
 		devl_lock(devlink);
-		err = devlink_reload(devlink, &init_net,
-				     DEVLINK_RELOAD_ACTION_DRIVER_REINIT,
-				     DEVLINK_RELOAD_LIMIT_UNSPEC,
-				     &actions_performed, NULL);
+		err = 0;
+		if (devl_is_alive(devlink))
+			err = devlink_reload(devlink, &init_net,
+					     DEVLINK_RELOAD_ACTION_DRIVER_REINIT,
+					     DEVLINK_RELOAD_LIMIT_UNSPEC,
+					     &actions_performed, NULL);
 		devl_unlock(devlink);
 		devlink_put(devlink);
 
net/devlink/netlink.c +8 -2
index b38df704be1c..773efaabb6ad 100644
@@ -98,7 +98,8 @@ devlink_get_from_attrs_lock(struct net *net, struct nlattr **attrs)
 
 	devlinks_xa_for_each_registered_get(net, index, devlink) {
 		devl_lock(devlink);
-		if (strcmp(devlink->dev->bus->name, busname) == 0 &&
+		if (devl_is_alive(devlink) &&
+		    strcmp(devlink->dev->bus->name, busname) == 0 &&
 		    strcmp(dev_name(devlink->dev), devname) == 0)
 			return devlink;
 		devl_unlock(devlink);
@@ -210,7 +211,12 @@ int devlink_instance_iter_dump(struct sk_buff *msg, struct netlink_callback *cb)
 
 	devlink_dump_for_each_instance_get(msg, dump, devlink) {
 		devl_lock(devlink);
-		err = cmd->dump_one(msg, devlink, cb);
+
+		if (devl_is_alive(devlink))
+			err = cmd->dump_one(msg, devlink, cb);
+		else
+			err = 0;
+
 		devl_unlock(devlink);
 		devlink_put(devlink);
 
Signature · 1 line
2.38.1
#11 lore raw

Re: [RFC net-next 04/10] devlink: always check if the devlink instance is registered

Quoted 3 lines · Jakub Kicinski

Always check under the instance lock whether the devlink instance is still / already registered.

Ok. So now the reference ensures less about whats valid. It guarantees a lock but doesn't ensure that the devlink remains registered unless you acquire the lock and check that the devlink is alive under lock now?

Quoted 6 lines

This is a no-op for the most part, as the unregistration path currently waits for all references. On the init path, however, we may temporarily open up a race with netdev code, if netdevs are registered before the devlink instance. This is temporary, the next change fixes it, and this commit has been split out for the ease of review.

This means you're adding the problem here, but its fixed in next commit..?

Quoted 4 lines

Note that in case of iterating over sub-objects which have their own lock (regions and line cards) we assume an implicit dependency between those objects existing and devlink unregistration.

That seems reasonable.

Quoted 188 lines
---
 include/net/devlink.h |  1 +
 net/devlink/basic.c   | 35 +++++++++++++++++++++++++++++------
 net/devlink/core.c    | 25 +++++++++++++++++++++----
 net/devlink/netlink.c | 10 ++++++++--
 4 files changed, 59 insertions(+), 12 deletions(-)
include/net/devlink.h +1 -0
index 6a2e4f21779f..36e013d3aa52 100644
@@ -1626,6 +1626,7 @@ struct device *devlink_to_dev(const struct devlink *devlink);
 void devl_lock(struct devlink *devlink);
 int devl_trylock(struct devlink *devlink);
 void devl_unlock(struct devlink *devlink);
+bool devl_is_alive(struct devlink *devlink);
 void devl_assert_locked(struct devlink *devlink);
 bool devl_lock_is_held(struct devlink *devlink);
 
net/devlink/basic.c +29 -6
index 5f33d74eef83..6b18e70a39fd 100644
@@ -2130,6 +2130,9 @@ static int devlink_nl_cmd_linecard_get_dumpit(struct sk_buff *msg,
 		int idx = 0;
 
 		mutex_lock(&devlink->linecards_lock);
+		if (!devl_is_alive(devlink))
+			goto next_devlink;
+
 		list_for_each_entry(linecard, &devlink->linecard_list, list) {
 			if (idx < dump->idx) {
 				idx++;
@@ -2151,6 +2154,7 @@ static int devlink_nl_cmd_linecard_get_dumpit(struct sk_buff *msg,
 			}
 			idx++;
 		}
+next_devlink:
 		mutex_unlock(&devlink->linecards_lock);
 		devlink_put(devlink);
 	}
@@ -7809,6 +7813,12 @@ devlink_nl_cmd_health_reporter_get_dumpit(struct sk_buff *msg,
 		int idx = 0;
 
 		mutex_lock(&devlink->reporters_lock);
+		if (!devl_is_alive(devlink)) {
+			mutex_unlock(&devlink->reporters_lock);
+			devlink_put(devlink);
+			continue;
+		}
+
 		list_for_each_entry(reporter, &devlink->reporter_list,
 				    list) {
 			if (idx < dump->idx) {
@@ -7830,6 +7840,9 @@ devlink_nl_cmd_health_reporter_get_dumpit(struct sk_buff *msg,
 		mutex_unlock(&devlink->reporters_lock);
 
 		devl_lock(devlink);
+		if (!devl_is_alive(devlink))
+			goto next_devlink;
+
 		xa_for_each(&devlink->ports, port_index, port) {
 			mutex_lock(&port->reporters_lock);
 			list_for_each_entry(reporter, &port->reporter_list, list) {
@@ -7853,6 +7866,7 @@ devlink_nl_cmd_health_reporter_get_dumpit(struct sk_buff *msg,
 			}
 			mutex_unlock(&port->reporters_lock);
 		}
+next_devlink:
 		devl_unlock(devlink);
 		devlink_put(devlink);
 	}
@@ -12218,7 +12232,8 @@ void devlink_compat_running_version(struct devlink *devlink,
 		return;
 
 	devl_lock(devlink);
-	__devlink_compat_running_version(devlink, buf, len);
+	if (devl_is_alive(devlink))
+		__devlink_compat_running_version(devlink, buf, len);
 	devl_unlock(devlink);
 }
 
@@ -12227,20 +12242,28 @@ int devlink_compat_flash_update(struct devlink *devlink, const char *file_name)
 	struct devlink_flash_update_params params = {};
 	int ret;
 
-	if (!devlink->ops->flash_update)
-		return -EOPNOTSUPP;
+	devl_lock(devlink);
+	if (!devl_is_alive(devlink)) {
+		ret = -ENODEV;
+		goto out_unlock;
+	}
+
+	if (!devlink->ops->flash_update) {
+		ret = -EOPNOTSUPP;
+		goto out_unlock;
+	}
 
 	ret = request_firmware(&params.fw, file_name, devlink->dev);
 	if (ret)
-		return ret;
+		goto out_unlock;
 
-	devl_lock(devlink);
 	devlink_flash_update_begin_notify(devlink);
 	ret = devlink->ops->flash_update(devlink, &params, NULL);
 	devlink_flash_update_end_notify(devlink);
-	devl_unlock(devlink);
 
 	release_firmware(params.fw);
+out_unlock:
+	devl_unlock(devlink);
 
 	return ret;
 }
net/devlink/core.c +21 -4
index d3b8336946fd..2abad8247597 100644
@@ -67,6 +67,21 @@ void devl_unlock(struct devlink *devlink)
 }
 EXPORT_SYMBOL_GPL(devl_unlock);
 
+bool devl_is_alive(struct devlink *devlink)
+{
+	return xa_get_mark(&devlinks, devlink->index, DEVLINK_REGISTERED);
+}
+EXPORT_SYMBOL_GPL(devl_is_alive);
+
+/**
+ * devlink_try_get() - try to obtain a reference on a devlink instance
+ * @devlink: instance to reference
+ *
+ * Obtain a reference on a devlink instance. A reference on a devlink instance
+ * only implies that it's safe to take the instance lock. It does not imply
+ * that the instance is registered, use devl_is_alive() after taking
+ * the instance lock to check registration status.
+ */
 struct devlink *__must_check devlink_try_get(struct devlink *devlink)
 {
 	if (refcount_inc_not_zero(&devlink->refcount))
@@ -300,10 +315,12 @@ static void __net_exit devlink_pernet_pre_exit(struct net *net)
 	devlinks_xa_for_each_registered_get(net, index, devlink) {
 		WARN_ON(!(devlink->features & DEVLINK_F_RELOAD));
 		devl_lock(devlink);
-		err = devlink_reload(devlink, &init_net,
-				     DEVLINK_RELOAD_ACTION_DRIVER_REINIT,
-				     DEVLINK_RELOAD_LIMIT_UNSPEC,
-				     &actions_performed, NULL);
+		err = 0;
+		if (devl_is_alive(devlink))
+			err = devlink_reload(devlink, &init_net,
+					     DEVLINK_RELOAD_ACTION_DRIVER_REINIT,
+					     DEVLINK_RELOAD_LIMIT_UNSPEC,
+					     &actions_performed, NULL);
 		devl_unlock(devlink);
 		devlink_put(devlink);
 
net/devlink/netlink.c +8 -2
index b38df704be1c..773efaabb6ad 100644
@@ -98,7 +98,8 @@ devlink_get_from_attrs_lock(struct net *net, struct nlattr **attrs)
 
 	devlinks_xa_for_each_registered_get(net, index, devlink) {
 		devl_lock(devlink);
-		if (strcmp(devlink->dev->bus->name, busname) == 0 &&
+		if (devl_is_alive(devlink) &&
+		    strcmp(devlink->dev->bus->name, busname) == 0 &&
 		    strcmp(dev_name(devlink->dev), devname) == 0)
 			return devlink;
 		devl_unlock(devlink);
@@ -210,7 +211,12 @@ int devlink_instance_iter_dump(struct sk_buff *msg, struct netlink_callback *cb)
 
 	devlink_dump_for_each_instance_get(msg, dump, devlink) {
 		devl_lock(devlink);
-		err = cmd->dump_one(msg, devlink, cb);
+
+		if (devl_is_alive(devlink))
+			err = cmd->dump_one(msg, devlink, cb);
+		else
+			err = 0;
+
 		devl_unlock(devlink);
 		devlink_put(devlink);
 
author maintainer
#12 lore raw

Re: [RFC net-next 04/10] devlink: always check if the devlink instance is registered

Quoted 7 lines · On Mon, 19 Dec 2022 09:48:54 -0800 Jacob Keller wrote:
Quoted 2 lines · Jakub Kicinski

Always check under the instance lock whether the devlink instance is still / already registered.

Ok. So now the reference ensures less about whats valid. It guarantees a lock but doesn't ensure that the devlink remains registered unless you acquire the lock and check that the devlink is alive under lock now?

Correct.

Quoted 8 lines
Quoted 6 lines

This is a no-op for the most part, as the unregistration path currently waits for all references. On the init path, however, we may temporarily open up a race with netdev code, if netdevs are registered before the devlink instance. This is temporary, the next change fixes it, and this commit has been split out for the ease of review.

This means you're adding the problem here, but its fixed in next commit..?

Yes, I can squash when posting for applying, but TBH I think the clarity of the changes outweighs the tiny and transient race.

#13 lore raw

Re: [RFC net-next 04/10] devlink: always check if the devlink instance is registered

Quoted 18 lines · Jakub Kicinski
Quoted 7 lines · On Mon, 19 Dec 2022 09:48:54 -0800 Jacob Keller wrote:
Quoted 2 lines · Jakub Kicinski

Always check under the instance lock whether the devlink instance is still / already registered.

Ok. So now the reference ensures less about whats valid. It guarantees a lock but doesn't ensure that the devlink remains registered unless you acquire the lock and check that the devlink is alive under lock now?

Correct.

Quoted 7 lines
Quoted 6 lines

This is a no-op for the most part, as the unregistration path currently waits for all references. On the init path, however, we may temporarily open up a race with netdev code, if netdevs are registered before the devlink instance. This is temporary, the next change fixes it, and this commit has been split out for the ease of review.

Quoted 4 lines
Quoted 1 line

This means you're adding the problem here, but its fixed in next commit..?

Yes, I can squash when posting for applying, but TBH I think the clarity of the changes outweighs the tiny and transient race.

I would agree. The only reason I could think it to be a problem is if a bisect lands precisely on this commit and you happen to hit this... what are the side effects of the race here? If the side effects don't include significant issues I think its fine.

maintainer
#14 lore raw

Re: [RFC net-next 04/10] devlink: always check if the devlink instance is registered

In reply to Jakub Kicinski
Quoted 9 lines · Sat, Dec 17, 2022 at 02:19:47AM CET, [email protected] wrote:

Always check under the instance lock whether the devlink instance is still / already registered.

This is a no-op for the most part, as the unregistration path currently waits for all references. On the init path, however, we may temporarily open up a race with netdev code, if netdevs are registered before the devlink instance. This is temporary, the next change fixes it, and this commit has been split out for the ease of review.

Quoted 3 lines

Note that in case of iterating over sub-objects which have their own lock (regions and line cards) we assume an implicit dependency between those objects existing and devlink unregistration.

This would be probably very valuable to add as a comment inside the code for the future reader mind sake.

Quoted 123 lines

---

include/net/devlink.h |  1 +
net/devlink/basic.c   | 35 +++++++++++++++++++++++++++++------
net/devlink/core.c    | 25 +++++++++++++++++++++----
net/devlink/netlink.c | 10 ++++++++--
4 files changed, 59 insertions(+), 12 deletions(-)
include/net/devlink.h +1 -0
index 6a2e4f21779f..36e013d3aa52 100644
@@ -1626,6 +1626,7 @@ struct device *devlink_to_dev(const struct devlink *devlink);
 void devl_lock(struct devlink *devlink);
 int devl_trylock(struct devlink *devlink);
 void devl_unlock(struct devlink *devlink);
+bool devl_is_alive(struct devlink *devlink);
 void devl_assert_locked(struct devlink *devlink);
 bool devl_lock_is_held(struct devlink *devlink);
 
net/devlink/basic.c +29 -6
index 5f33d74eef83..6b18e70a39fd 100644
@@ -2130,6 +2130,9 @@ static int devlink_nl_cmd_linecard_get_dumpit(struct sk_buff *msg,
 		int idx = 0;
 
 		mutex_lock(&devlink->linecards_lock);
+		if (!devl_is_alive(devlink))
+			goto next_devlink;
+
 		list_for_each_entry(linecard, &devlink->linecard_list, list) {
 			if (idx < dump->idx) {
 				idx++;
@@ -2151,6 +2154,7 @@ static int devlink_nl_cmd_linecard_get_dumpit(struct sk_buff *msg,
 			}
 			idx++;
 		}
+next_devlink:
 		mutex_unlock(&devlink->linecards_lock);
 		devlink_put(devlink);
 	}
@@ -7809,6 +7813,12 @@ devlink_nl_cmd_health_reporter_get_dumpit(struct sk_buff *msg,
 		int idx = 0;
 
 		mutex_lock(&devlink->reporters_lock);
+		if (!devl_is_alive(devlink)) {
+			mutex_unlock(&devlink->reporters_lock);
+			devlink_put(devlink);
+			continue;
+		}
+
 		list_for_each_entry(reporter, &devlink->reporter_list,
 				    list) {
 			if (idx < dump->idx) {
@@ -7830,6 +7840,9 @@ devlink_nl_cmd_health_reporter_get_dumpit(struct sk_buff *msg,
 		mutex_unlock(&devlink->reporters_lock);
 
 		devl_lock(devlink);
+		if (!devl_is_alive(devlink))
+			goto next_devlink;
+
 		xa_for_each(&devlink->ports, port_index, port) {
 			mutex_lock(&port->reporters_lock);
 			list_for_each_entry(reporter, &port->reporter_list, list) {
@@ -7853,6 +7866,7 @@ devlink_nl_cmd_health_reporter_get_dumpit(struct sk_buff *msg,
 			}
 			mutex_unlock(&port->reporters_lock);
 		}
+next_devlink:
 		devl_unlock(devlink);
 		devlink_put(devlink);
 	}
@@ -12218,7 +12232,8 @@ void devlink_compat_running_version(struct devlink *devlink,
 		return;
 
 	devl_lock(devlink);
-	__devlink_compat_running_version(devlink, buf, len);
+	if (devl_is_alive(devlink))
+		__devlink_compat_running_version(devlink, buf, len);
 	devl_unlock(devlink);
 }
 
@@ -12227,20 +12242,28 @@ int devlink_compat_flash_update(struct devlink *devlink, const char *file_name)
 	struct devlink_flash_update_params params = {};
 	int ret;
 
-	if (!devlink->ops->flash_update)
-		return -EOPNOTSUPP;
+	devl_lock(devlink);
+	if (!devl_is_alive(devlink)) {
+		ret = -ENODEV;
+		goto out_unlock;
+	}
+
+	if (!devlink->ops->flash_update) {
+		ret = -EOPNOTSUPP;
+		goto out_unlock;
+	}
 
 	ret = request_firmware(&params.fw, file_name, devlink->dev);
 	if (ret)
-		return ret;
+		goto out_unlock;
 
-	devl_lock(devlink);
 	devlink_flash_update_begin_notify(devlink);
 	ret = devlink->ops->flash_update(devlink, &params, NULL);
 	devlink_flash_update_end_notify(devlink);
-	devl_unlock(devlink);
 
 	release_firmware(params.fw);
+out_unlock:
+	devl_unlock(devlink);
 
 	return ret;
 }
net/devlink/core.c +0 -0
index d3b8336946fd..2abad8247597 100644

net/devlink/core.c void devl_unlock(struct devlink *devlink) · line 70

Quoted 4 lines
net/devlink/core.c +1 -0
@@ -67,6 +67,21 @@ void devl_unlock(struct devlink *devlink)
 }
 EXPORT_SYMBOL_GPL(devl_unlock);
 
+bool devl_is_alive(struct devlink *devlink)

Why "alive"? To be consistent with the existing terminology, how about to name it devl_is_registered()?

Also, "devl_" implicates that it should be called with devlink instance lock held, so probably devlink_is_registered() would be better.

Quoted 4 lines
net/devlink/core.c +4 -0
+{
+	return xa_get_mark(&devlinks, devlink->index, DEVLINK_REGISTERED);
+}
+EXPORT_SYMBOL_GPL(devl_is_alive);

net/devlink/core.c void devl_unlock(struct devlink *devlink) · line 84

Quoted 10 lines
net/devlink/core.c +10 -0
+
+/**
+ * devlink_try_get() - try to obtain a reference on a devlink instance
+ * @devlink: instance to reference
+ *
+ * Obtain a reference on a devlink instance. A reference on a devlink instance
+ * only implies that it's safe to take the instance lock. It does not imply
+ * that the instance is registered, use devl_is_alive() after taking
+ * the instance lock to check registration status.
+ */

This comment is not related to the patch, should be added in a separate one.

Quoted 51 lines
net/devlink/core.c +6 -4
 struct devlink *__must_check devlink_try_get(struct devlink *devlink)
 {
 	if (refcount_inc_not_zero(&devlink->refcount))
@@ -300,10 +315,12 @@ static void __net_exit devlink_pernet_pre_exit(struct net *net)
 	devlinks_xa_for_each_registered_get(net, index, devlink) {
 		WARN_ON(!(devlink->features & DEVLINK_F_RELOAD));
 		devl_lock(devlink);
-		err = devlink_reload(devlink, &init_net,
-				     DEVLINK_RELOAD_ACTION_DRIVER_REINIT,
-				     DEVLINK_RELOAD_LIMIT_UNSPEC,
-				     &actions_performed, NULL);
+		err = 0;
+		if (devl_is_alive(devlink))
+			err = devlink_reload(devlink, &init_net,
+					     DEVLINK_RELOAD_ACTION_DRIVER_REINIT,
+					     DEVLINK_RELOAD_LIMIT_UNSPEC,
+					     &actions_performed, NULL);
 		devl_unlock(devlink);
 		devlink_put(devlink);
 
net/devlink/netlink.c +8 -2
index b38df704be1c..773efaabb6ad 100644
@@ -98,7 +98,8 @@ devlink_get_from_attrs_lock(struct net *net, struct nlattr **attrs)
 
 	devlinks_xa_for_each_registered_get(net, index, devlink) {
 		devl_lock(devlink);
-		if (strcmp(devlink->dev->bus->name, busname) == 0 &&
+		if (devl_is_alive(devlink) &&
+		    strcmp(devlink->dev->bus->name, busname) == 0 &&
 		    strcmp(dev_name(devlink->dev), devname) == 0)
 			return devlink;
 		devl_unlock(devlink);
@@ -210,7 +211,12 @@ int devlink_instance_iter_dump(struct sk_buff *msg, struct netlink_callback *cb)
 
 	devlink_dump_for_each_instance_get(msg, dump, devlink) {
 		devl_lock(devlink);
-		err = cmd->dump_one(msg, devlink, cb);
+
+		if (devl_is_alive(devlink))
+			err = cmd->dump_one(msg, devlink, cb);
+		else
+			err = 0;
+
 		devl_unlock(devlink);
 		devlink_put(devlink);
 
Signature · 1 line
2.38.1
author maintainer
#15 lore raw

Re: [RFC net-next 04/10] devlink: always check if the devlink instance is registered

Quoted 10 lines · On Mon, 2 Jan 2023 14:58:16 +0100 Jiri Pirko wrote:
Quoted 9 lines · Sat, Dec 17, 2022 at 02:19:47AM CET, [email protected] wrote:

Always check under the instance lock whether the devlink instance is still / already registered.

This is a no-op for the most part, as the unregistration path currently waits for all references. On the init path, however, we may temporarily open up a race with netdev code, if netdevs are registered before the devlink instance. This is temporary, the next change fixes it, and this commit has been split out for the ease of review.

Quoted 6 lines
Quoted 3 lines

Note that in case of iterating over sub-objects which have their own lock (regions and line cards) we assume an implicit dependency between those objects existing and devlink unregistration.

This would be probably very valuable to add as a comment inside the code for the future reader mind sake.

Where, tho?

I'm strongly against the pointlessly fine-grained locking going forward so hopefully there won't be any more per-subobject locks added anyway.

Quoted 4 lines
Quoted 1 line

+bool devl_is_alive(struct devlink *devlink)

Why "alive"? To be consistent with the existing terminology, how about to name it devl_is_registered()?

I dislike the similarity to device_is_registered() which has very different semantics. I prefer alive.

Quoted 2 lines

Also, "devl_" implicates that it should be called with devlink instance lock held, so probably devlink_is_registered() would be better.

I'm guessing you realized this isn't correct later on.

Quoted 9 lines
Quoted 9 lines
net/devlink/core.c +9 -0
+{
+	return xa_get_mark(&devlinks, devlink->index, DEVLINK_REGISTERED);
+}
+EXPORT_SYMBOL_GPL(devl_is_alive);
+
+/**
+ * devlink_try_get() - try to obtain a reference on a devlink instance
+ * @devlink: instance to reference
+ *
Quoted 8 lines
Quoted 5 lines

+ * Obtain a reference on a devlink instance. A reference on a devlink instance
+ * only implies that it's safe to take the instance lock. It does not imply
+ * that the instance is registered, use devl_is_alive() after taking
+ * the instance lock to check registration status.
+ */

This comment is not related to the patch, should be added in a separate one.

The point of adding this comment is to say that one has to use devl_is_alive() after accessing an instance by reference. It is very much in the right patch.

maintainer
#16 lore raw

Re: [RFC net-next 04/10] devlink: always check if the devlink instance is registered

Quoted 15 lines · Tue, Jan 03, 2023 at 12:05:14AM CET, [email protected] wrote:
Quoted 14 lines · On Mon, 2 Jan 2023 14:58:16 +0100 Jiri Pirko wrote:
Quoted 12 lines · Sat, Dec 17, 2022 at 02:19:47AM CET, [email protected] wrote:

Always check under the instance lock whether the devlink instance is still / already registered.

This is a no-op for the most part, as the unregistration path currently waits for all references. On the init path, however, we may temporarily open up a race with netdev code, if netdevs are registered before the devlink instance. This is temporary, the next change fixes it, and this commit has been split out for the ease of review.

Note that in case of iterating over sub-objects which have their own lock (regions and line cards) we assume an implicit dependency between those objects existing and devlink unregistration.

Quoted 7 lines
Quoted 2 lines

This would be probably very valuable to add as a comment inside the code for the future reader mind sake.

Where, tho?

I'm strongly against the pointlessly fine-grained locking going forward so hopefully there won't be any more per-subobject locks added anyway.

Agreed. That is what I suggested in the other thread too.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

+bool devl_is_alive(struct devlink *devlink)

Why "alive"? To be consistent with the existing terminology, how about to name it devl_is_registered()?

I dislike the similarity to device_is_registered() which has very different semantics. I prefer alive.

Interesting. Didn't occur to me to look into device.h when reading devlink.c code. I mean, is device_register() behaviour in sync with devlink_register?

Your alive() helper is checking "register mark". It's an odd and unneded inconsistency in newly added code :/

Quoted 5 lines
Quoted 3 lines

Also, "devl_" implicates that it should be called with devlink instance lock held, so probably devlink_is_registered() would be better.

I'm guessing you realized this isn't correct later on.

From what I see, no need to hold instance mutex for xa mark checking, alhough I understand why you want the helper to be called with the lock. Perhaps assert and a little comment would make this clear?

Quoted 19 lines
Quoted 10 lines
Quoted 10 lines
net/devlink/core.c +9 -0
+{
+	return xa_get_mark(&devlinks, devlink->index, DEVLINK_REGISTERED);
+}
+EXPORT_SYMBOL_GPL(devl_is_alive);
+
+/**
+ * devlink_try_get() - try to obtain a reference on a devlink instance
+ * @devlink: instance to reference
+ *
Quoted 8 lines
Quoted 5 lines

+ * Obtain a reference on a devlink instance. A reference on a devlink instance
+ * only implies that it's safe to take the instance lock. It does not imply
+ * that the instance is registered, use devl_is_alive() after taking
+ * the instance lock to check registration status.
+ */

This comment is not related to the patch, should be added in a separate one.

Quoted 3 lines

The point of adding this comment is to say that one has to use devl_is_alive() after accessing an instance by reference. It is very much in the right patch.

Gotha! My mistake, sorry.

author maintainer
#17 lore raw

Re: [RFC net-next 04/10] devlink: always check if the devlink instance is registered

Quoted 6 lines · On Tue, 3 Jan 2023 10:26:12 +0100 Jiri Pirko wrote:
Quoted 5 lines
Quoted 2 lines

Why "alive"? To be consistent with the existing terminology, how about to name it devl_is_registered()?

I dislike the similarity to device_is_registered() which has very different semantics. I prefer alive.

Quoted 6 lines

Interesting. Didn't occur to me to look into device.h when reading devlink.c code. I mean, is device_register() behaviour in sync with devlink_register?

Your alive() helper is checking "register mark". It's an odd and unneded inconsistency in newly added code :/

Alright.

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

Also, "devl_" implicates that it should be called with devlink instance lock held, so probably devlink_is_registered() would be better.

I'm guessing you realized this isn't correct later on.

From what I see, no need to hold instance mutex for xa mark checking, alhough I understand why you want the helper to be called with the lock. Perhaps assert and a little comment would make this clear?

I'll add the comment. The assert would have to OR holding the subobject locks. Is that what you had in mind?

maintainer
#18 lore raw

Re: [RFC net-next 04/10] devlink: always check if the devlink instance is registered

Quoted 19 lines · Wed, Jan 04, 2023 at 03:49:59AM CET, [email protected] wrote:
Quoted 6 lines · On Tue, 3 Jan 2023 10:26:12 +0100 Jiri Pirko wrote:
Quoted 5 lines
Quoted 2 lines

Why "alive"? To be consistent with the existing terminology, how about to name it devl_is_registered()?

I dislike the similarity to device_is_registered() which has very different semantics. I prefer alive.

Quoted 6 lines

Interesting. Didn't occur to me to look into device.h when reading devlink.c code. I mean, is device_register() behaviour in sync with devlink_register?

Your alive() helper is checking "register mark". It's an odd and unneded inconsistency in newly added code :/

Alright.

Quoted 3 lines
Quoted 3 lines
Quoted 2 lines

Also, "devl_" implicates that it should be called with devlink instance lock held, so probably devlink_is_registered() would be better.

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

I'm guessing you realized this isn't correct later on.

From what I see, no need to hold instance mutex for xa mark checking, alhough I understand why you want the helper to be called with the lock. Perhaps assert and a little comment would make this clear?

I'll add the comment. The assert would have to OR holding the subobject locks. Is that what you had in mind?

Ah right, the subobject locks. That will go away I'm sure. Yes, that is what I had in mind. After that, we can put assert here.

maintainer
#19 lore raw

Re: [RFC net-next 04/10] devlink: always check if the devlink instance is registered

In reply to Jakub Kicinski
Quoted 38 lines · Sat, Dec 17, 2022 at 02:19:47AM CET, [email protected] wrote:

Always check under the instance lock whether the devlink instance is still / already registered.

This is a no-op for the most part, as the unregistration path currently waits for all references. On the init path, however, we may temporarily open up a race with netdev code, if netdevs are registered before the devlink instance. This is temporary, the next change fixes it, and this commit has been split out for the ease of review.

Note that in case of iterating over sub-objects which have their own lock (regions and line cards) we assume an implicit dependency between those objects existing and devlink unregistration.

---

include/net/devlink.h |  1 +
net/devlink/basic.c   | 35 +++++++++++++++++++++++++++++------
net/devlink/core.c    | 25 +++++++++++++++++++++----
net/devlink/netlink.c | 10 ++++++++--
4 files changed, 59 insertions(+), 12 deletions(-)
include/net/devlink.h +1 -0
index 6a2e4f21779f..36e013d3aa52 100644
@@ -1626,6 +1626,7 @@ struct device *devlink_to_dev(const struct devlink *devlink);
 void devl_lock(struct devlink *devlink);
 int devl_trylock(struct devlink *devlink);
 void devl_unlock(struct devlink *devlink);
+bool devl_is_alive(struct devlink *devlink);
 void devl_assert_locked(struct devlink *devlink);
 bool devl_lock_is_held(struct devlink *devlink);
 
net/devlink/basic.c +0 -0
index 5f33d74eef83..6b18e70a39fd 100644

net/devlink/basic.c static int devlink_nl_cmd_linecard_get_dumpit(struct sk_buff *msg, · line 2134

Quoted 5 lines
net/devlink/basic.c +2 -0
@@ -2130,6 +2130,9 @@ static int devlink_nl_cmd_linecard_get_dumpit(struct sk_buff *msg,
 		int idx = 0;
 
 		mutex_lock(&devlink->linecards_lock);
+		if (!devl_is_alive(devlink))
+			goto next_devlink;

Thinking about this a bit more, things would be cleaner if reporters and linecards are converted to rely on instance lock as well. I don't see a good reason for a separate lock in both cases, really.

Also, we could introduce devlinks_xa_for_each_registered_get_lock() iterator that would lock the instance as well right away to avoid this devl_is_alive() dance on multiple places when you iterate devlinks.

Quoted 44 lines
net/devlink/basic.c +12 -0
+
 		list_for_each_entry(linecard, &devlink->linecard_list, list) {
 			if (idx < dump->idx) {
 				idx++;
@@ -2151,6 +2154,7 @@ static int devlink_nl_cmd_linecard_get_dumpit(struct sk_buff *msg,
 			}
 			idx++;
 		}
+next_devlink:
 		mutex_unlock(&devlink->linecards_lock);
 		devlink_put(devlink);
 	}
@@ -7809,6 +7813,12 @@ devlink_nl_cmd_health_reporter_get_dumpit(struct sk_buff *msg,
 		int idx = 0;
 
 		mutex_lock(&devlink->reporters_lock);
+		if (!devl_is_alive(devlink)) {
+			mutex_unlock(&devlink->reporters_lock);
+			devlink_put(devlink);
+			continue;
+		}
+
 		list_for_each_entry(reporter, &devlink->reporter_list,
 				    list) {
 			if (idx < dump->idx) {
@@ -7830,6 +7840,9 @@ devlink_nl_cmd_health_reporter_get_dumpit(struct sk_buff *msg,
 		mutex_unlock(&devlink->reporters_lock);
 
 		devl_lock(devlink);
+		if (!devl_is_alive(devlink))
+			goto next_devlink;
+
 		xa_for_each(&devlink->ports, port_index, port) {
 			mutex_lock(&port->reporters_lock);
 			list_for_each_entry(reporter, &port->reporter_list, list) {
@@ -7853,6 +7866,7 @@ devlink_nl_cmd_health_reporter_get_dumpit(struct sk_buff *msg,
 			}
 			mutex_unlock(&port->reporters_lock);
 		}
+next_devlink:
 		devl_unlock(devlink);
 		devlink_put(devlink);
 	}

net/devlink/basic.c void devlink_compat_running_version(struct devlink *devlink, · line 12234

Quoted 3 lines
net/devlink/basic.c +0 -0
@@ -12218,7 +12232,8 @@ void devlink_compat_running_version(struct devlink *devlink,
 		return;
 
 	devl_lock(devlink);

How about to have a helper, something like devl_lock_alive() (or devl_lock_registered() with the naming scheme I suggest in the other thread)? Then you can do:

	if (!devl_lock_alive(devlink))
		return;
	__devlink_compat_running_version(devlink, buf, len);
	devl_unlock(devlink);
Quoted 114 lines
net/devlink/basic.c +15 -6
-	__devlink_compat_running_version(devlink, buf, len);
+	if (devl_is_alive(devlink))
+		__devlink_compat_running_version(devlink, buf, len);
 	devl_unlock(devlink);
 }
 
@@ -12227,20 +12242,28 @@ int devlink_compat_flash_update(struct devlink *devlink, const char *file_name)
 	struct devlink_flash_update_params params = {};
 	int ret;
 
-	if (!devlink->ops->flash_update)
-		return -EOPNOTSUPP;
+	devl_lock(devlink);
+	if (!devl_is_alive(devlink)) {
+		ret = -ENODEV;
+		goto out_unlock;
+	}
+
+	if (!devlink->ops->flash_update) {
+		ret = -EOPNOTSUPP;
+		goto out_unlock;
+	}
 
 	ret = request_firmware(&params.fw, file_name, devlink->dev);
 	if (ret)
-		return ret;
+		goto out_unlock;
 
-	devl_lock(devlink);
 	devlink_flash_update_begin_notify(devlink);
 	ret = devlink->ops->flash_update(devlink, &params, NULL);
 	devlink_flash_update_end_notify(devlink);
-	devl_unlock(devlink);
 
 	release_firmware(params.fw);
+out_unlock:
+	devl_unlock(devlink);
 
 	return ret;
 }
net/devlink/core.c +21 -4
index d3b8336946fd..2abad8247597 100644
@@ -67,6 +67,21 @@ void devl_unlock(struct devlink *devlink)
 }
 EXPORT_SYMBOL_GPL(devl_unlock);
 
+bool devl_is_alive(struct devlink *devlink)
+{
+	return xa_get_mark(&devlinks, devlink->index, DEVLINK_REGISTERED);
+}
+EXPORT_SYMBOL_GPL(devl_is_alive);
+
+/**
+ * devlink_try_get() - try to obtain a reference on a devlink instance
+ * @devlink: instance to reference
+ *
+ * Obtain a reference on a devlink instance. A reference on a devlink instance
+ * only implies that it's safe to take the instance lock. It does not imply
+ * that the instance is registered, use devl_is_alive() after taking
+ * the instance lock to check registration status.
+ */
 struct devlink *__must_check devlink_try_get(struct devlink *devlink)
 {
 	if (refcount_inc_not_zero(&devlink->refcount))
@@ -300,10 +315,12 @@ static void __net_exit devlink_pernet_pre_exit(struct net *net)
 	devlinks_xa_for_each_registered_get(net, index, devlink) {
 		WARN_ON(!(devlink->features & DEVLINK_F_RELOAD));
 		devl_lock(devlink);
-		err = devlink_reload(devlink, &init_net,
-				     DEVLINK_RELOAD_ACTION_DRIVER_REINIT,
-				     DEVLINK_RELOAD_LIMIT_UNSPEC,
-				     &actions_performed, NULL);
+		err = 0;
+		if (devl_is_alive(devlink))
+			err = devlink_reload(devlink, &init_net,
+					     DEVLINK_RELOAD_ACTION_DRIVER_REINIT,
+					     DEVLINK_RELOAD_LIMIT_UNSPEC,
+					     &actions_performed, NULL);
 		devl_unlock(devlink);
 		devlink_put(devlink);
 
net/devlink/netlink.c +8 -2
index b38df704be1c..773efaabb6ad 100644
@@ -98,7 +98,8 @@ devlink_get_from_attrs_lock(struct net *net, struct nlattr **attrs)
 
 	devlinks_xa_for_each_registered_get(net, index, devlink) {
 		devl_lock(devlink);
-		if (strcmp(devlink->dev->bus->name, busname) == 0 &&
+		if (devl_is_alive(devlink) &&
+		    strcmp(devlink->dev->bus->name, busname) == 0 &&
 		    strcmp(dev_name(devlink->dev), devname) == 0)
 			return devlink;
 		devl_unlock(devlink);
@@ -210,7 +211,12 @@ int devlink_instance_iter_dump(struct sk_buff *msg, struct netlink_callback *cb)
 
 	devlink_dump_for_each_instance_get(msg, dump, devlink) {
 		devl_lock(devlink);
-		err = cmd->dump_one(msg, devlink, cb);
+
+		if (devl_is_alive(devlink))
+			err = cmd->dump_one(msg, devlink, cb);
+		else
+			err = 0;
+
 		devl_unlock(devlink);
 		devlink_put(devlink);
 
Signature · 1 line
2.38.1
maintainer
#20 lore raw

Re: [RFC net-next 04/10] devlink: always check if the devlink instance is registered

Quoted 50 lines · Mon, Jan 02, 2023 at 03:57:24PM CET, [email protected] wrote:
Quoted 38 lines · Sat, Dec 17, 2022 at 02:19:47AM CET, [email protected] wrote:

Always check under the instance lock whether the devlink instance is still / already registered.

This is a no-op for the most part, as the unregistration path currently waits for all references. On the init path, however, we may temporarily open up a race with netdev code, if netdevs are registered before the devlink instance. This is temporary, the next change fixes it, and this commit has been split out for the ease of review.

Note that in case of iterating over sub-objects which have their own lock (regions and line cards) we assume an implicit dependency between those objects existing and devlink unregistration.

---

include/net/devlink.h |  1 +
net/devlink/basic.c   | 35 +++++++++++++++++++++++++++++------
net/devlink/core.c    | 25 +++++++++++++++++++++----
net/devlink/netlink.c | 10 ++++++++--
4 files changed, 59 insertions(+), 12 deletions(-)
include/net/devlink.h +1 -0
index 6a2e4f21779f..36e013d3aa52 100644
@@ -1626,6 +1626,7 @@ struct device *devlink_to_dev(const struct devlink *devlink);
 void devl_lock(struct devlink *devlink);
 int devl_trylock(struct devlink *devlink);
 void devl_unlock(struct devlink *devlink);
+bool devl_is_alive(struct devlink *devlink);
 void devl_assert_locked(struct devlink *devlink);
 bool devl_lock_is_held(struct devlink *devlink);
 
net/devlink/basic.c +0 -0
index 5f33d74eef83..6b18e70a39fd 100644

net/devlink/basic.c static int devlink_nl_cmd_linecard_get_dumpit(struct sk_buff *msg, · line 2134

Quoted 5 lines
net/devlink/basic.c +2 -0
@@ -2130,6 +2130,9 @@ static int devlink_nl_cmd_linecard_get_dumpit(struct sk_buff *msg,
 		int idx = 0;
 
 		mutex_lock(&devlink->linecards_lock);
+		if (!devl_is_alive(devlink))
+			goto next_devlink;

Thinking about this a bit more, things would be cleaner if reporters and linecards are converted to rely on instance lock as well. I don't see a good reason for a separate lock in both cases, really.

Quoted 3 lines

Also, we could introduce devlinks_xa_for_each_registered_get_lock() iterator that would lock the instance as well right away to avoid this devl_is_alive() dance on multiple places when you iterate devlinks.

devlinks_xa_find_get_locked()
would do the check&lock at the end.

Quoted 6 lines
Quoted 6 lines
net/devlink/basic.c +1 -0
+
 		list_for_each_entry(linecard, &devlink->linecard_list, list) {
 			if (idx < dump->idx) {
 				idx++;

[...]

author maintainer
#21 lore raw

Re: [RFC net-next 04/10] devlink: always check if the devlink instance is registered

In reply to Jiri Pirko
Quoted 21 lines · On Mon, 2 Jan 2023 15:57:24 +0100 Jiri Pirko wrote:
Quoted 20 lines · Sat, Dec 17, 2022 at 02:19:47AM CET, [email protected] wrote:

Always check under the instance lock whether the devlink instance is still / already registered.

This is a no-op for the most part, as the unregistration path currently waits for all references. On the init path, however, we may temporarily open up a race with netdev code, if netdevs are registered before the devlink instance. This is temporary, the next change fixes it, and this commit has been split out for the ease of review.

Note that in case of iterating over sub-objects which have their own lock (regions and line cards) we assume an implicit dependency between those objects existing and devlink unregistration.

net/devlink/basic.c +0 -0
index 5f33d74eef83..6b18e70a39fd 100644
@@ -2130,6 +2130,9 @@ static int devlink_nl_cmd_linecard_get_dumpit(struct sk_buff *msg,
 		int idx = 0;
 
Quoted 7 lines

net/devlink/basic.c static int devlink_nl_cmd_linecard_get_dumpit(struct sk_buff *msg, · line 2134

Quoted 3 lines
net/devlink/basic.c +2 -0
 		mutex_lock(&devlink->linecards_lock);
+		if (!devl_is_alive(devlink))
+			goto next_devlink;  

Thinking about this a bit more, things would be cleaner if reporters and linecards are converted to rely on instance lock as well. I don't see a good reason for a separate lock in both cases, really.

We had discussion before, I'm pretty sure.
IIRC you said that mlx4's locking prevents us from using the instance lock for regions.

Quoted 3 lines

Also, we could introduce devlinks_xa_for_each_registered_get_lock() iterator that would lock the instance as well right away to avoid this devl_is_alive() dance on multiple places when you iterate devlinks.

That's what I started with, but the ability to factor our the unlock/put on error paths made the callback approach much cleaner. And after using the callback for all the dumps there's only a couple places which would use devlinks_xa_for_each_registered_get_lock().

Quoted 9 lines

net/devlink/basic.c void devlink_compat_running_version(struct devlink *devlink, · line 12234

Quoted 4 lines
net/devlink/basic.c +0 -0
@@ -12218,7 +12232,8 @@ void devlink_compat_running_version(struct devlink *devlink,
 		return;
 
 	devl_lock(devlink);  

How about to have a helper, something like devl_lock_alive() (or devl_lock_registered() with the naming scheme I suggest in the other thread)? Then you can do:

Quoted 4 lines
	if (!devl_lock_alive(devlink))
		return;
	__devlink_compat_running_version(devlink, buf, len);
	devl_unlock(devlink);

I guess aesthetic preference.

If I had the cycles I'd make devlink_try_get() return a wrapped type

struct devlink_ref {
	struct devlink *devlink;
};

which one would have to pass to devl_lock_from_ref() or some such:

struct devlink *devl_lock_from_ref(struct devlink_ref dref)

{
	if (!dref.devlink)
		return NULL;
	devl_lock(dref.devlink);
	if (devl_lock_alive(dref.devlink))
		return dref.devlink;
	devl_unlock(dref.devlink);
	return NULL;
}

But the number of calls to devl_is_alive() is quite small after all the cleanup, so I don't think the extra helpers are justified at this point. "Normal coders" should not be exposed to any of the lifetime details, not when coding the drivers, not when adding typical devlink features/subobjects.

maintainer
#22 lore raw

Re: [RFC net-next 04/10] devlink: always check if the devlink instance is registered

Quoted 30 lines · Tue, Jan 03, 2023 at 12:16:30AM CET, [email protected] wrote:
Quoted 21 lines · On Mon, 2 Jan 2023 15:57:24 +0100 Jiri Pirko wrote:
Quoted 20 lines · Sat, Dec 17, 2022 at 02:19:47AM CET, [email protected] wrote:

Always check under the instance lock whether the devlink instance is still / already registered.

This is a no-op for the most part, as the unregistration path currently waits for all references. On the init path, however, we may temporarily open up a race with netdev code, if netdevs are registered before the devlink instance. This is temporary, the next change fixes it, and this commit has been split out for the ease of review.

Note that in case of iterating over sub-objects which have their own lock (regions and line cards) we assume an implicit dependency between those objects existing and devlink unregistration.

net/devlink/basic.c +0 -0
index 5f33d74eef83..6b18e70a39fd 100644
@@ -2130,6 +2130,9 @@ static int devlink_nl_cmd_linecard_get_dumpit(struct sk_buff *msg,
 		int idx = 0;
 
Quoted 7 lines

net/devlink/basic.c static int devlink_nl_cmd_linecard_get_dumpit(struct sk_buff *msg, · line 2134

Quoted 3 lines
net/devlink/basic.c +2 -0
 		mutex_lock(&devlink->linecards_lock);
+		if (!devl_is_alive(devlink))
+			goto next_devlink;  

Thinking about this a bit more, things would be cleaner if reporters and linecards are converted to rely on instance lock as well. I don't see a good reason for a separate lock in both cases, really.

Quoted 3 lines

We had discussion before, I'm pretty sure.
IIRC you said that mlx4's locking prevents us from using the instance lock for regions.

Yeah, let me check it out again. For the linecards, that could be done. Let me take care of these.

Quoted 9 lines
Quoted 4 lines

Also, we could introduce devlinks_xa_for_each_registered_get_lock() iterator that would lock the instance as well right away to avoid this devl_is_alive() dance on multiple places when you iterate devlinks.

That's what I started with, but the ability to factor our the unlock/put on error paths made the callback approach much cleaner. And after using the callback for all the dumps there's only a couple places which would use devlinks_xa_for_each_registered_get_lock().

I see. Okay.

Quoted 36 lines
Quoted 10 lines

net/devlink/basic.c void devlink_compat_running_version(struct devlink *devlink, · line 12234

Quoted 5 lines
net/devlink/basic.c +0 -0
@@ -12218,7 +12232,8 @@ void devlink_compat_running_version(struct devlink *devlink,
 		return;
 
 	devl_lock(devlink);  

How about to have a helper, something like devl_lock_alive() (or devl_lock_registered() with the naming scheme I suggest in the other thread)? Then you can do:

Quoted 4 lines
	if (!devl_lock_alive(devlink))
		return;
	__devlink_compat_running_version(devlink, buf, len);
	devl_unlock(devlink);

I guess aesthetic preference.

If I had the cycles I'd make devlink_try_get() return a wrapped type

struct devlink_ref {
	struct devlink *devlink;
};

which one would have to pass to devl_lock_from_ref() or some such:

struct devlink *devl_lock_from_ref(struct devlink_ref dref)

{
	if (!dref.devlink)
		return NULL;
	devl_lock(dref.devlink);
	if (devl_lock_alive(dref.devlink))
		return dref.devlink;
	devl_unlock(dref.devlink);
	return NULL;
}
Quoted 5 lines

But the number of calls to devl_is_alive() is quite small after all the cleanup, so I don't think the extra helpers are justified at this point. "Normal coders" should not be exposed to any of the lifetime details, not when coding the drivers, not when adding typical devlink features/subobjects.

Fair point.

maintainer
#23 lore raw

Re: [RFC net-next 04/10] devlink: always check if the devlink instance is registered

In reply to Jakub Kicinski

Sat, Dec 17, 2022 at 02:19:47AM CET, [email protected] wrote:

[...]

net/devlink/core.c void devl_unlock(struct devlink *devlink) · line 74

Quoted 5 lines
net/devlink/core.c +5 -0
+bool devl_is_alive(struct devlink *devlink)
+{
+	return xa_get_mark(&devlinks, devlink->index, DEVLINK_REGISTERED);
+}
+EXPORT_SYMBOL_GPL(devl_is_alive);

Why is this exported? Drivers should not use this, as you said.

author maintainer
#24 lore raw

Re: [RFC net-next 04/10] devlink: always check if the devlink instance is registered

Quoted 7 lines · On Tue, 3 Jan 2023 13:26:30 +0100 Jiri Pirko wrote:

net/devlink/core.c void devl_unlock(struct devlink *devlink) · line 74

Quoted 5 lines
net/devlink/core.c +5 -0
+bool devl_is_alive(struct devlink *devlink)
+{
+	return xa_get_mark(&devlinks, devlink->index, DEVLINK_REGISTERED);
+}
+EXPORT_SYMBOL_GPL(devl_is_alive);  

Why is this exported? Drivers should not use this, as you said.

I'll make it a static inline in the internal header.

author maintainer
#25 lore raw

RFC 5/10 devlink: remove the registration guarantee of references

In reply to Jakub Kicinski

The objective of exposing the devlink instance locks to drivers was to let them use these locks to prevent user space from accessing the device before it's fully initialized. This is difficult because devlink_unregister() waits for all references to be released, meaning that devlink_unregister() can't itself be called under the instance lock.

To avoid this issue devlink_register() was moved after subobject registration a while ago. Unfortunately the netdev paths get a hold of the devlink instances _before_ they are registered. Ideally netdev should wait for devlink init to finish (synchronizing on the instance lock). This can't work because we don't know if the instance will _ever_ be registered (in case of failures it may not). The other option of returning an error until devlink_register() is called is unappealing (user space would get a notification netdev exist but would have to wait arbitrary amount of time before accessing some of its attributes).

Weaken the guarantees of the devlink references.

Holding a reference will now only guarantee that the memory of the object is around. Another way of looking at it is that the reference now protects the object not its "registered" status. Use devlink instance lock to synchronize unregistration.

This implies that releasing of the "main" reference of the devlink instance moves from devlink_unregister() to devlink_free().

---
 include/net/devlink.h       |  2 ++
 net/devlink/core.c          | 64 ++++++++++++++++---------------------
 net/devlink/devl_internal.h |  2 --
 3 files changed, 30 insertions(+), 38 deletions(-)
include/net/devlink.h +2 -0
index 36e013d3aa52..cc910612b3f4 100644
@@ -1648,6 +1648,8 @@ static inline struct devlink *devlink_alloc(const struct devlink_ops *ops,
 	return devlink_alloc_ns(ops, priv_size, &init_net, dev);
 }
 void devlink_set_features(struct devlink *devlink, u64 features);
+int devl_register(struct devlink *devlink);
+void devl_unregister(struct devlink *devlink);
 void devlink_register(struct devlink *devlink);
 void devlink_unregister(struct devlink *devlink);
 void devlink_free(struct devlink *devlink);
net/devlink/core.c +28 -36

113 lines not on this page · raw message

net/devlink/devl_internal.h +0 -2
index c3977c69552a..7e77eebde3b9 100644
@@ -12,7 +12,6 @@
 #include <net/net_namespace.h>
 
 #define DEVLINK_REGISTERED XA_MARK_1
-#define DEVLINK_UNREGISTERING XA_MARK_2
 
 #define DEVLINK_RELOAD_STATS_ARRAY_SIZE \
 	(__DEVLINK_RELOAD_LIMIT_MAX * __DEVLINK_RELOAD_ACTION_MAX)
@@ -52,7 +51,6 @@ struct devlink {
 	struct lock_class_key lock_key;
 	u8 reload_failed:1;
 	refcount_t refcount;
-	struct completion comp;
 	struct rcu_head rcu;
 	struct notifier_block netdevice_nb;
 	char priv[] __aligned(NETDEV_ALIGN);
Signature · 1 line
2.38.1
Reviewed-by on v1 5/10
#26 lore raw

Re: [RFC net-next 05/10] devlink: remove the registration guarantee of references

Quoted 7 lines · Jakub Kicinski

The objective of exposing the devlink instance locks to drivers was to let them use these locks to prevent user space from accessing the device before it's fully initialized. This is difficult because devlink_unregister() waits for all references to be released, meaning that devlink_unregister() can't itself be called under the instance lock.

Sure.

Quoted 11 lines

To avoid this issue devlink_register() was moved after subobject registration a while ago. Unfortunately the netdev paths get a hold of the devlink instances _before_ they are registered. Ideally netdev should wait for devlink init to finish (synchronizing on the instance lock). This can't work because we don't know if the instance will _ever_ be registered (in case of failures it may not). The other option of returning an error until devlink_register() is called is unappealing (user space would get a notification netdev exist but would have to wait arbitrary amount of time before accessing some of its attributes).

Nice summary of the problems and options that we have tried already.

I think its also important as this can allow sub objects to be registered after the devlink instance?

Quoted 7 lines

Weaken the guarantees of the devlink references.

Holding a reference will now only guarantee that the memory of the object is around. Another way of looking at it is that the reference now protects the object not its "registered" status. Use devlink instance lock to synchronize unregistration.

Right, this makes sense.

Quoted 3 lines

This implies that releasing of the "main" reference of the devlink instance moves from devlink_unregister() to devlink_free().

This makes sense and I think aligns more with how most references work in practice. Good.

Quoted 1 line

Code change seems straight forward enough. I had a minor question, but:

Quoted 71 lines
---
 include/net/devlink.h       |  2 ++
 net/devlink/core.c          | 64 ++++++++++++++++---------------------
 net/devlink/devl_internal.h |  2 --
 3 files changed, 30 insertions(+), 38 deletions(-)
include/net/devlink.h +2 -0
index 36e013d3aa52..cc910612b3f4 100644
@@ -1648,6 +1648,8 @@ static inline struct devlink *devlink_alloc(const struct devlink_ops *ops,
 	return devlink_alloc_ns(ops, priv_size, &init_net, dev);
 }
 void devlink_set_features(struct devlink *devlink, u64 features);
+int devl_register(struct devlink *devlink);
+void devl_unregister(struct devlink *devlink);
 void devlink_register(struct devlink *devlink);
 void devlink_unregister(struct devlink *devlink);
 void devlink_free(struct devlink *devlink);
net/devlink/core.c +3 -23
index 2abad8247597..413b92534ad6 100644
@@ -89,21 +89,10 @@ struct devlink *__must_check devlink_try_get(struct devlink *devlink)
 	return NULL;
 }
 
-static void __devlink_put_rcu(struct rcu_head *head)
-{
-	struct devlink *devlink = container_of(head, struct devlink, rcu);
-
-	complete(&devlink->comp);
-}
-
 void devlink_put(struct devlink *devlink)
 {
 	if (refcount_dec_and_test(&devlink->refcount))
-		/* Make sure unregister operation that may await the completion
-		 * is unblocked only after all users are after the end of
-		 * RCU grace period.
-		 */
-		call_rcu(&devlink->rcu, __devlink_put_rcu);
+		kfree_rcu(devlink, rcu);
 }
 
 struct devlink *devlinks_xa_find_get(struct net *net, unsigned long *indexp)
@@ -116,13 +105,6 @@ struct devlink *devlinks_xa_find_get(struct net *net, unsigned long *indexp)
 	if (!devlink)
 		goto unlock;
 
-	/* In case devlink_unregister() was already called and "unregistering"
-	 * mark was set, do not allow to get a devlink reference here.
-	 * This prevents live-lock of devlink_unregister() wait for completion.
-	 */
-	if (xa_get_mark(&devlinks, *indexp, DEVLINK_UNREGISTERING))
-		goto next;
-
 	if (!devlink_try_get(devlink))
 		goto next;
 	if (!net_eq(devlink_net(devlink), net)) {
@@ -158,37 +140,48 @@ void devlink_set_features(struct devlink *devlink, u64 features)
 EXPORT_SYMBOL_GPL(devlink_set_features);
 
 /**
- *	devlink_register - Register devlink instance
- *
- *	@devlink: devlink
+ * devl_register - Register devlink instance
+ * @devlink: devlink
  */
-void devlink_register(struct devlink *devlink)

net/devlink/core.c void devlink_set_features(struct devlink *devlink, u64 features) · line 154

Quoted 10 lines
net/devlink/core.c +4 -1
+int devl_register(struct devlink *devlink)
 {
 	ASSERT_DEVLINK_NOT_REGISTERED(devlink);
-	/* Make sure that we are in .probe() routine */
+	devl_assert_locked(devlink);
 
 	xa_set_mark(&devlinks, devlink->index, DEVLINK_REGISTERED);
 	devlink_notify_register(devlink);
+
+	return 0;

Any particular reason to change this to int when it doesn't have a failure case yet? Future patches I assume? You don't check the devl_register return value.

Quoted 80 lines
net/devlink/core.c +21 -12
+}
+EXPORT_SYMBOL_GPL(devl_register);
+
+void devlink_register(struct devlink *devlink)
+{
+	devl_lock(devlink);
+	devl_register(devlink);
+	devl_unlock(devlink);
 }
 EXPORT_SYMBOL_GPL(devlink_register);
 
 /**
- *	devlink_unregister - Unregister devlink instance
- *
- *	@devlink: devlink
+ * devl_unregister - Unregister devlink instance
+ * @devlink: devlink
  */
-void devlink_unregister(struct devlink *devlink)
+void devl_unregister(struct devlink *devlink)
 {
 	ASSERT_DEVLINK_REGISTERED(devlink);
-	/* Make sure that we are in .remove() routine */
-
-	xa_set_mark(&devlinks, devlink->index, DEVLINK_UNREGISTERING);
-	devlink_put(devlink);
-	wait_for_completion(&devlink->comp);
+	devl_assert_locked(devlink);
 
 	devlink_notify_unregister(devlink);
 	xa_clear_mark(&devlinks, devlink->index, DEVLINK_REGISTERED);
-	xa_clear_mark(&devlinks, devlink->index, DEVLINK_UNREGISTERING);
+}
+EXPORT_SYMBOL_GPL(devl_unregister);
+
+void devlink_unregister(struct devlink *devlink)
+{
+	devl_lock(devlink);
+	devl_unregister(devlink);
+	devl_unlock(devlink);
 }
 EXPORT_SYMBOL_GPL(devlink_unregister);
 
@@ -252,7 +245,6 @@ struct devlink *devlink_alloc_ns(const struct devlink_ops *ops,
 	mutex_init(&devlink->reporters_lock);
 	mutex_init(&devlink->linecards_lock);
 	refcount_set(&devlink->refcount, 1);
-	init_completion(&devlink->comp);
 
 	return devlink;
 
@@ -298,7 +290,7 @@ void devlink_free(struct devlink *devlink)
 
 	xa_erase(&devlinks, devlink->index);
 
-	kfree(devlink);
+	devlink_put(devlink);
 }
 EXPORT_SYMBOL_GPL(devlink_free);
 
net/devlink/devl_internal.h +0 -2
index c3977c69552a..7e77eebde3b9 100644
@@ -12,7 +12,6 @@
 #include <net/net_namespace.h>
 
 #define DEVLINK_REGISTERED XA_MARK_1
-#define DEVLINK_UNREGISTERING XA_MARK_2
 
 #define DEVLINK_RELOAD_STATS_ARRAY_SIZE \
 	(__DEVLINK_RELOAD_LIMIT_MAX * __DEVLINK_RELOAD_ACTION_MAX)
@@ -52,7 +51,6 @@ struct devlink {
 	struct lock_class_key lock_key;
 	u8 reload_failed:1;
 	refcount_t refcount;
-	struct completion comp;
 	struct rcu_head rcu;
 	struct notifier_block netdevice_nb;
 	char priv[] __aligned(NETDEV_ALIGN);
author maintainer
#27 lore raw

Re: [RFC net-next 05/10] devlink: remove the registration guarantee of references

Quoted 7 lines · On Mon, 19 Dec 2022 09:56:26 -0800 Jacob Keller wrote:
Quoted 7 lines
net/devlink/core.c +2 -2
-void devlink_register(struct devlink *devlink)
+int devl_register(struct devlink *devlink)
 {
 	ASSERT_DEVLINK_NOT_REGISTERED(devlink);
-	/* Make sure that we are in .probe() routine */
+	devl_assert_locked(devlink);
Quoted 8 lines

net/devlink/core.c void devlink_set_features(struct devlink *devlink, u64 features) · line 154

Quoted 4 lines
net/devlink/core.c +2 -0
 	xa_set_mark(&devlinks, devlink->index, DEVLINK_REGISTERED);
 	devlink_notify_register(devlink);
+
+	return 0;  

Any particular reason to change this to int when it doesn't have a failure case yet? Future patches I assume? You don't check the devl_register return value.

I was wondering if anyone would notice :)

Returning errors from the registration helper seems natural, and if we don't have this ability it may impact our ability to extend the core in the long run.
I was against making core functions void in the first place. It's a good opportunity to change back.

#28 lore raw

Re: [RFC net-next 05/10] devlink: remove the registration guarantee of references

Quoted 17 lines · Jakub Kicinski
Quoted 7 lines · On Mon, 19 Dec 2022 09:56:26 -0800 Jacob Keller wrote:
Quoted 7 lines
net/devlink/core.c +2 -2
-void devlink_register(struct devlink *devlink)
+int devl_register(struct devlink *devlink)
 {
 	ASSERT_DEVLINK_NOT_REGISTERED(devlink);
-	/* Make sure that we are in .probe() routine */
+	devl_assert_locked(devlink);
Quoted 8 lines

net/devlink/core.c void devlink_set_features(struct devlink *devlink, u64 features) · line 154

Quoted 4 lines
net/devlink/core.c +2 -0
 	xa_set_mark(&devlinks, devlink->index, DEVLINK_REGISTERED);
 	devlink_notify_register(devlink);
+
+	return 0;  

Any particular reason to change this to int when it doesn't have a failure case yet? Future patches I assume? You don't check the devl_register return value.

Quoted 2 lines

I was wondering if anyone would notice :)

I'm fine with it, but I would expect that devlink_register would want to report it at least?

Quoted 5 lines

Returning errors from the registration helper seems natural, and if we don't have this ability it may impact our ability to extend the core in the long run.
I was against making core functions void in the first place. It's a good opportunity to change back.

Sure. I think its better to be able to report an error but wanted to make sure its actually caught or at least logged if it occurs.

We can ofcourse change the function templates again since we don't really guarantee API stability across versions, but it is more work for backporting in the future.

author maintainer
#29 lore raw

Re: [RFC net-next 05/10] devlink: remove the registration guarantee of references

Quoted 5 lines · On Mon, 19 Dec 2022 14:14:18 -0800 Jacob Keller wrote:
Quoted 1 line · Jakub Kicinski

I was wondering if anyone would notice :)

I'm fine with it, but I would expect that devlink_register would want to report it at least?

New code should not use devlink_* functions, so probably not worth it.

Quoted 12 lines
Quoted 5 lines

Returning errors from the registration helper seems natural, and if we don't have this ability it may impact our ability to extend the core in the long run.
I was against making core functions void in the first place. It's a good opportunity to change back.

Sure. I think its better to be able to report an error but wanted to make sure its actually caught or at least logged if it occurs.

We can ofcourse change the function templates again since we don't really guarantee API stability across versions, but it is more work for backporting in the future.

maintainer
#30 lore raw

Re: [RFC net-next 05/10] devlink: remove the registration guarantee of references

In reply to Jakub Kicinski
Quoted 17 lines · Mon, Dec 19, 2022 at 11:02:10PM CET, [email protected] wrote:
Quoted 7 lines · On Mon, 19 Dec 2022 09:56:26 -0800 Jacob Keller wrote:
Quoted 7 lines
net/devlink/core.c +2 -2
-void devlink_register(struct devlink *devlink)
+int devl_register(struct devlink *devlink)
 {
 	ASSERT_DEVLINK_NOT_REGISTERED(devlink);
-	/* Make sure that we are in .probe() routine */
+	devl_assert_locked(devlink);
Quoted 8 lines

net/devlink/core.c void devlink_set_features(struct devlink *devlink, u64 features) · line 154

Quoted 4 lines
net/devlink/core.c +2 -0
 	xa_set_mark(&devlinks, devlink->index, DEVLINK_REGISTERED);
 	devlink_notify_register(devlink);
+
+	return 0;  

Any particular reason to change this to int when it doesn't have a failure case yet? Future patches I assume? You don't check the devl_register return value.

Quoted 7 lines

I was wondering if anyone would notice :)

Returning errors from the registration helper seems natural, and if we don't have this ability it may impact our ability to extend the core in the long run.
I was against making core functions void in the first place. It's a good opportunity to change back.

devlink_register originally returned int. Leon changed that as part of his work. I believe I expressed my negative feelings about that back then. Sigh.

maintainer
#31 lore raw

Re: [RFC net-next 05/10] devlink: remove the registration guarantee of references

In reply to Jakub Kicinski
Quoted 1 line · Sat, Dec 17, 2022 at 02:19:48AM CET, [email protected] wrote:

[...]

Quoted 3 lines

Holding a reference will now only guarantee that the memory of the object is around. Another way of looking at it is that the reference now protects the object not its "registered" status.

This would help to understand what you are doing in patch 3. Perphaps it would be fine to describe the goal in the cover letter?

author maintainer
#32 lore raw

Re: [RFC net-next 05/10] devlink: remove the registration guarantee of references

Quoted 6 lines · On Mon, 2 Jan 2023 15:32:04 +0100 Jiri Pirko wrote:
Quoted 3 lines

Holding a reference will now only guarantee that the memory of the object is around. Another way of looking at it is that the reference now protects the object not its "registered" status.

This would help to understand what you are doing in patch 3. Perphaps it would be fine to describe the goal in the cover letter?

Fair point, will do!

author maintainer
#33 lore raw

RFC 6/10 devlink: don't require setting features before registration

In reply to Jakub Kicinski

Requiring devlink_set_features() to be run before devlink is registered is overzealous. devlink_set_features() itself is a leftover from old workarounds which were trying to prevent initiating reload before probe was complete.

---
 net/devlink/core.c | 2 --
 1 file changed, 2 deletions(-)
net/devlink/core.c +0 -2
index 413b92534ad6..f30fc167c8ad 100644
@@ -131,8 +131,6 @@ struct devlink *devlinks_xa_find_get(struct net *net, unsigned long *indexp)
  */
 void devlink_set_features(struct devlink *devlink, u64 features)
 {
-	ASSERT_DEVLINK_NOT_REGISTERED(devlink);
-
 	WARN_ON(features & DEVLINK_F_RELOAD &&
 		!devlink_reload_supported(devlink->ops));
 	devlink->features = features;
Signature · 1 line
2.38.1
maintainer
#34 lore raw

Re: [RFC net-next 06/10] devlink: don't require setting features before registration

Quoted 4 lines · Sat, Dec 17, 2022 at 02:19:49AM CET, [email protected] wrote:

Requiring devlink_set_features() to be run before devlink is registered is overzealous. devlink_set_features() itself is a leftover from old workarounds which were trying to prevent initiating reload before probe was complete.

Wouldn't it be better to remove this entirely? I don't think it is needed anymore.

Quoted 22 lines

---
net/devlink/core.c | 2 --

1 file changed, 2 deletions(-)
net/devlink/core.c +0 -2
index 413b92534ad6..f30fc167c8ad 100644
@@ -131,8 +131,6 @@ struct devlink *devlinks_xa_find_get(struct net *net, unsigned long *indexp)
 */
 void devlink_set_features(struct devlink *devlink, u64 features)
 {
-	ASSERT_DEVLINK_NOT_REGISTERED(devlink);
-
 	WARN_ON(features & DEVLINK_F_RELOAD &&
 		!devlink_reload_supported(devlink->ops));
 	devlink->features = features;
Signature · 1 line
2.38.1
author maintainer
#35 lore raw

Re: [RFC net-next 06/10] devlink: don't require setting features before registration

Quoted 8 lines · On Mon, 2 Jan 2023 16:25:18 +0100 Jiri Pirko wrote:
Quoted 4 lines · Sat, Dec 17, 2022 at 02:19:49AM CET, [email protected] wrote:

Requiring devlink_set_features() to be run before devlink is registered is overzealous. devlink_set_features() itself is a leftover from old workarounds which were trying to prevent initiating reload before probe was complete.

Wouldn't it be better to remove this entirely? I don't think it is needed anymore.

I think you're right. Since users don't have access to the instance before it's registered - this flag can have no impact.

author maintainer
#36 lore raw

Re: [RFC net-next 06/10] devlink: don't require setting features before registration

Quoted 7 lines · On Mon, 2 Jan 2023 15:24:47 -0800 Jakub Kicinski wrote:
Quoted 6 lines · On Mon, 2 Jan 2023 16:25:18 +0100 Jiri Pirko wrote:
Quoted 4 lines · Sat, Dec 17, 2022 at 02:19:49AM CET, [email protected] wrote:

Requiring devlink_set_features() to be run before devlink is registered is overzealous. devlink_set_features() itself is a leftover from old workarounds which were trying to prevent initiating reload before probe was complete.

Quoted 5 lines
Quoted 2 lines

Wouldn't it be better to remove this entirely? I don't think it is needed anymore.

I think you're right. Since users don't have access to the instance before it's registered - this flag can have no impact.

Let's leave this for a separate follow up, mlx5 needs a bit more work. It sets the feature conditionally.

maintainer
#37 lore raw

Re: [RFC net-next 06/10] devlink: don't require setting features before registration

Quoted 8 lines · Tue, Jan 03, 2023 at 12:32:54AM CET, [email protected] wrote:
Quoted 7 lines · On Mon, 2 Jan 2023 15:24:47 -0800 Jakub Kicinski wrote:
Quoted 6 lines · On Mon, 2 Jan 2023 16:25:18 +0100 Jiri Pirko wrote:
Quoted 4 lines · Sat, Dec 17, 2022 at 02:19:49AM CET, [email protected] wrote:

Requiring devlink_set_features() to be run before devlink is registered is overzealous. devlink_set_features() itself is a leftover from old workarounds which were trying to prevent initiating reload before probe was complete.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Wouldn't it be better to remove this entirely? I don't think it is needed anymore.

I think you're right. Since users don't have access to the instance before it's registered - this flag can have no impact.

Let's leave this for a separate follow up, mlx5 needs a bit more work. It sets the feature conditionally.

Okay, let me take care of that.

author maintainer
#38 lore raw

RFC 7/10 netdevsim: rename a label

In reply to Jakub Kicinski

err_dl_unregister should unregister the devlink instance. Looks like renaming it was missed in one of the reshufflings.

---
 drivers/net/netdevsim/dev.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/net/netdevsim/dev.c +2 -2
index b962fc8e1397..d25f6e86d901 100644
@@ -1563,7 +1563,7 @@ int nsim_drv_probe(struct nsim_bus_dev *nsim_bus_dev)
 	err = devlink_params_register(devlink, nsim_devlink_params,
 				      ARRAY_SIZE(nsim_devlink_params));
 	if (err)
-		goto err_dl_unregister;
+		goto err_resource_unregister;
 	nsim_devlink_set_params_init_values(nsim_dev, devlink);
 
 	err = nsim_dev_dummy_region_init(nsim_dev, devlink);
@@ -1629,7 +1629,7 @@ int nsim_drv_probe(struct nsim_bus_dev *nsim_bus_dev)
 err_params_unregister:
 	devlink_params_unregister(devlink, nsim_devlink_params,
 				  ARRAY_SIZE(nsim_devlink_params));
-err_dl_unregister:
+err_resource_unregister:
 	devl_resources_unregister(devlink);
 err_vfc_free:
 	kfree(nsim_dev->vfconfigs);
Signature · 1 line
2.38.1
Reviewed-by on v1 7/10
#39 lore raw

Re: [RFC net-next 07/10] netdevsim: rename a label

Quoted 4 lines · Jakub Kicinski

err_dl_unregister should unregister the devlink instance. Looks like renaming it was missed in one of the reshufflings.

Makes sense.

Quoted 26 lines
---
 drivers/net/netdevsim/dev.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/net/netdevsim/dev.c +2 -2
index b962fc8e1397..d25f6e86d901 100644
@@ -1563,7 +1563,7 @@ int nsim_drv_probe(struct nsim_bus_dev *nsim_bus_dev)
 	err = devlink_params_register(devlink, nsim_devlink_params,
 				      ARRAY_SIZE(nsim_devlink_params));
 	if (err)
-		goto err_dl_unregister;
+		goto err_resource_unregister;
 	nsim_devlink_set_params_init_values(nsim_dev, devlink);
 
 	err = nsim_dev_dummy_region_init(nsim_dev, devlink);
@@ -1629,7 +1629,7 @@ int nsim_drv_probe(struct nsim_bus_dev *nsim_bus_dev)
 err_params_unregister:
 	devlink_params_unregister(devlink, nsim_devlink_params,
 				  ARRAY_SIZE(nsim_devlink_params));
-err_dl_unregister:
+err_resource_unregister:
 	devl_resources_unregister(devlink);
 err_vfc_free:
 	kfree(nsim_dev->vfconfigs);
author maintainer
#40 lore raw

RFC 8/10 netdevsim: move devlink registration under the instance lock

In reply to Jakub Kicinski

To prevent races with netdev code accessing free devlink instances move the registration under the devlink instance lock. Core now waits for the instance to be registered before accessing it.

---
 drivers/net/netdevsim/dev.c | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)
drivers/net/netdevsim/dev.c +8 -3
index d25f6e86d901..c9952a34c39a 100644
@@ -1566,10 +1566,14 @@ int nsim_drv_probe(struct nsim_bus_dev *nsim_bus_dev)
 		goto err_resource_unregister;
 	nsim_devlink_set_params_init_values(nsim_dev, devlink);
 
-	err = nsim_dev_dummy_region_init(nsim_dev, devlink);
+	err = devl_register(devlink);
 	if (err)
 		goto err_params_unregister;
 
+	err = nsim_dev_dummy_region_init(nsim_dev, devlink);
+	if (err)
+		goto err_dl_unregister;
+
 	err = nsim_dev_traps_init(devlink);
 	if (err)
 		goto err_dummy_region_exit;
@@ -1607,7 +1611,6 @@ int nsim_drv_probe(struct nsim_bus_dev *nsim_bus_dev)
 	nsim_dev->esw_mode = DEVLINK_ESWITCH_MODE_LEGACY;
 	devlink_set_features(devlink, DEVLINK_F_RELOAD);
 	devl_unlock(devlink);
-	devlink_register(devlink);
 	return 0;
 
 err_hwstats_exit:
@@ -1626,6 +1629,8 @@ int nsim_drv_probe(struct nsim_bus_dev *nsim_bus_dev)
 	nsim_dev_traps_exit(devlink);
 err_dummy_region_exit:
 	nsim_dev_dummy_region_exit(nsim_dev);
+err_dl_unregister:
+	devl_unregister(devlink);
 err_params_unregister:
 	devlink_params_unregister(devlink, nsim_devlink_params,
 				  ARRAY_SIZE(nsim_devlink_params));
@@ -1668,12 +1673,12 @@ void nsim_drv_remove(struct nsim_bus_dev *nsim_bus_dev)
 	struct nsim_dev *nsim_dev = dev_get_drvdata(&nsim_bus_dev->dev);
 	struct devlink *devlink = priv_to_devlink(nsim_dev);
 
-	devlink_unregister(devlink);
 	devl_lock(devlink);
 	nsim_dev_reload_destroy(nsim_dev);
 
 	nsim_bpf_dev_exit(nsim_dev);
 	nsim_dev_debugfs_exit(nsim_dev);
+	devl_unregister(devlink);
 	devlink_params_unregister(devlink, nsim_devlink_params,
 				  ARRAY_SIZE(nsim_devlink_params));
 	devl_resources_unregister(devlink);
Signature · 1 line
2.38.1
author maintainer
#41 lore raw

RFC 9/10 devlink: allow registering parameters after the instance

In reply to Jakub Kicinski

It's most natural to register the instance first and then its subobjects. Now that we can use the instance lock to protect the atomicity of all init - it should also be safe.

---
 net/devlink/basic.c | 22 +++++++++++-----------
 1 file changed, 11 insertions(+), 11 deletions(-)
net/devlink/basic.c +11 -11
index 6b18e70a39fd..10b90a49aba0 100644
@@ -5263,7 +5263,13 @@ static void devlink_param_notify(struct devlink *devlink,
 	WARN_ON(cmd != DEVLINK_CMD_PARAM_NEW && cmd != DEVLINK_CMD_PARAM_DEL &&
 		cmd != DEVLINK_CMD_PORT_PARAM_NEW &&
 		cmd != DEVLINK_CMD_PORT_PARAM_DEL);
-	ASSERT_DEVLINK_REGISTERED(devlink);
+
+	/* devlink_notify_register() / devlink_notify_unregister()
+	 * will replay the notifications if the params are added/removed
+	 * outside of the lifetime of the instance.
+	 */
+	if (!devl_is_alive(devlink))
+		return;
 
 	msg = nlmsg_new(NLMSG_DEFAULT_SIZE, GFP_KERNEL);
 	if (!msg)
@@ -10915,8 +10921,6 @@ int devlink_params_register(struct devlink *devlink,
 	const struct devlink_param *param = params;
 	int i, err;
 
-	ASSERT_DEVLINK_NOT_REGISTERED(devlink);
-
 	for (i = 0; i < params_count; i++, param++) {
 		err = devlink_param_register(devlink, param);
 		if (err)
@@ -10947,8 +10951,6 @@ void devlink_params_unregister(struct devlink *devlink,
 	const struct devlink_param *param = params;
 	int i;
 
-	ASSERT_DEVLINK_NOT_REGISTERED(devlink);
-
 	for (i = 0; i < params_count; i++, param++)
 		devlink_param_unregister(devlink, param);
 }
@@ -10968,8 +10970,6 @@ int devlink_param_register(struct devlink *devlink,
 {
 	struct devlink_param_item *param_item;
 
-	ASSERT_DEVLINK_NOT_REGISTERED(devlink);
-
 	WARN_ON(devlink_param_verify(param));
 	WARN_ON(devlink_param_find_by_name(&devlink->param_list, param->name));
 
@@ -10985,6 +10985,7 @@ int devlink_param_register(struct devlink *devlink,
 	param_item->param = param;
 
 	list_add_tail(&param_item->list, &devlink->param_list);
+	devlink_param_notify(devlink, 0, param_item, DEVLINK_CMD_PARAM_NEW);
 	return 0;
 }
 EXPORT_SYMBOL_GPL(devlink_param_register);
@@ -10999,11 +11000,10 @@ void devlink_param_unregister(struct devlink *devlink,
 {
 	struct devlink_param_item *param_item;
 
-	ASSERT_DEVLINK_NOT_REGISTERED(devlink);
-
 	param_item =
 		devlink_param_find_by_name(&devlink->param_list, param->name);
 	WARN_ON(!param_item);
+	devlink_param_notify(devlink, 0, param_item, DEVLINK_CMD_PARAM_DEL);
 	list_del(&param_item->list);
 	kfree(param_item);
 }
@@ -11063,8 +11063,6 @@ int devlink_param_driverinit_value_set(struct devlink *devlink, u32 param_id,
 {
 	struct devlink_param_item *param_item;
 
-	ASSERT_DEVLINK_NOT_REGISTERED(devlink);
-
 	param_item = devlink_param_find_by_id(&devlink->param_list, param_id);
 	if (!param_item)
 		return -EINVAL;
@@ -11078,6 +11076,8 @@ int devlink_param_driverinit_value_set(struct devlink *devlink, u32 param_id,
 	else
 		param_item->driverinit_value = init_val;
 	param_item->driverinit_value_valid = true;
+
+	devlink_param_notify(devlink, 0, param_item, DEVLINK_CMD_PARAM_NEW);
 	return 0;
 }
 EXPORT_SYMBOL_GPL(devlink_param_driverinit_value_set);
Signature · 1 line
2.38.1
author maintainer
#42 lore raw

RFC 10/10 netdevsim: register devlink instance before sub-objects

In reply to Jakub Kicinski

Move the devlink instance registration up so that all the sub-object manipulation happens on a valid instance.

---
 drivers/net/netdevsim/dev.c | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)
drivers/net/netdevsim/dev.c +9 -9
index c9952a34c39a..738784fda117 100644
@@ -1556,23 +1556,23 @@ int nsim_drv_probe(struct nsim_bus_dev *nsim_bus_dev)
 		goto err_devlink_unlock;
 	}
 
-	err = nsim_dev_resources_register(devlink);
+	err = devl_register(devlink);
 	if (err)
 		goto err_vfc_free;
 
+	err = nsim_dev_resources_register(devlink);
+	if (err)
+		goto err_dl_unregister;
+
 	err = devlink_params_register(devlink, nsim_devlink_params,
 				      ARRAY_SIZE(nsim_devlink_params));
 	if (err)
 		goto err_resource_unregister;
 	nsim_devlink_set_params_init_values(nsim_dev, devlink);
 
-	err = devl_register(devlink);
-	if (err)
-		goto err_params_unregister;
-
 	err = nsim_dev_dummy_region_init(nsim_dev, devlink);
 	if (err)
-		goto err_dl_unregister;
+		goto err_params_unregister;
 
 	err = nsim_dev_traps_init(devlink);
 	if (err)
@@ -1629,13 +1629,13 @@ int nsim_drv_probe(struct nsim_bus_dev *nsim_bus_dev)
 	nsim_dev_traps_exit(devlink);
 err_dummy_region_exit:
 	nsim_dev_dummy_region_exit(nsim_dev);
-err_dl_unregister:
-	devl_unregister(devlink);
 err_params_unregister:
 	devlink_params_unregister(devlink, nsim_devlink_params,
 				  ARRAY_SIZE(nsim_devlink_params));
 err_resource_unregister:
 	devl_resources_unregister(devlink);
+err_dl_unregister:
+	devl_unregister(devlink);
 err_vfc_free:
 	kfree(nsim_dev->vfconfigs);
 err_devlink_unlock:
@@ -1678,10 +1678,10 @@ void nsim_drv_remove(struct nsim_bus_dev *nsim_bus_dev)
 
 	nsim_bpf_dev_exit(nsim_dev);
 	nsim_dev_debugfs_exit(nsim_dev);
-	devl_unregister(devlink);
 	devlink_params_unregister(devlink, nsim_devlink_params,
 				  ARRAY_SIZE(nsim_devlink_params));
 	devl_resources_unregister(devlink);
+	devl_unregister(devlink);
 	kfree(nsim_dev->vfconfigs);
 	kfree(nsim_dev->fa_cookie);
 	devl_unlock(devlink);
Signature · 1 line
2.38.1
maintainer
#43 lore raw

Re: [RFC net-next 10/10] netdevsim: register devlink instance before sub-objects

Quoted 2 lines · Sat, Dec 17, 2022 at 02:19:53AM CET, [email protected] wrote:

Move the devlink instance registration up so that all the sub-object manipulation happens on a valid instance.

I wonder, why don't you squash patch 8 to this one and make 1 move, to the fina destination?

author maintainer
#44 lore raw

Re: [RFC net-next 10/10] netdevsim: register devlink instance before sub-objects

Quoted 6 lines · On Mon, 2 Jan 2023 14:34:42 +0100 Jiri Pirko wrote:
Quoted 2 lines · Sat, Dec 17, 2022 at 02:19:53AM CET, [email protected] wrote:

Move the devlink instance registration up so that all the sub-object manipulation happens on a valid instance.

I wonder, why don't you squash patch 8 to this one and make 1 move, to the fina destination?

I found the squashed version a lot harder to review.

maintainer
#45 lore raw

Re: [RFC net-next 10/10] netdevsim: register devlink instance before sub-objects

Quoted 9 lines · Tue, Jan 03, 2023 at 12:25:46AM CET, [email protected] wrote:
Quoted 6 lines · On Mon, 2 Jan 2023 14:34:42 +0100 Jiri Pirko wrote:
Quoted 2 lines · Sat, Dec 17, 2022 at 02:19:53AM CET, [email protected] wrote:

Move the devlink instance registration up so that all the sub-object manipulation happens on a valid instance.

I wonder, why don't you squash patch 8 to this one and make 1 move, to the fina destination?

I found the squashed version a lot harder to review.

I'm puzzled. Both patches move calls to devl_register/unregister(). The first one moves it, the second one moves it a bit more. What's making the squashed patch hard to review?

author maintainer
#46 lore raw

Re: [RFC net-next 10/10] netdevsim: register devlink instance before sub-objects

Quoted 8 lines · On Tue, 3 Jan 2023 10:51:14 +0100 Jiri Pirko wrote:
Quoted 4 lines
Quoted 2 lines

I wonder, why don't you squash patch 8 to this one and make 1 move, to the fina destination?

I found the squashed version a lot harder to review.

I'm puzzled. Both patches move calls to devl_register/unregister(). The first one moves it, the second one moves it a bit more. What's making the squashed patch hard to review?

Ah, I thought you meant patch 7, sorry.
This one matters less, I'll squash.

#47 lore raw
In reply to Jakub Kicinski
Quoted 6 lines · Jakub Kicinski

This set is on top of the previous RFC.

Move the registration and unregistration of the devlink instances under their instance locks. Don't perform the netdev-style wait for all references when unregistering the instance.

Could you explain the reasoning/benefits here? I'm sure some of this is explained in each commit message as well but it would help to understand the overall series.

Quoted 20 lines

Jakub Kicinski (10):
devlink: bump the instance index directly when iterating
devlink: update the code in netns move to latest helpers
devlink: protect devlink->dev by the instance lock
devlink: always check if the devlink instance is registered
devlink: remove the registration guarantee of references
devlink: don't require setting features before registration
netdevsim: rename a label
netdevsim: move devlink registration under the instance lock
devlink: allow registering parameters after the instance
netdevsim: register devlink instance before sub-objects

 drivers/net/netdevsim/dev.c |  15 +++--
 include/net/devlink.h       |   3 +
 net/devlink/basic.c         |  64 ++++++++++++------
 net/devlink/core.c          | 127 +++++++++++++++++-------------------
 net/devlink/devl_internal.h |  20 ++----
 net/devlink/netlink.c       |  19 ++++--
 6 files changed, 136 insertions(+), 112 deletions(-)
author maintainer
#48 lore raw
Quoted 10 lines · On Mon, 19 Dec 2022 09:38:09 -0800 Jacob Keller wrote:
Quoted 5 lines · Jakub Kicinski

This set is on top of the previous RFC.

Move the registration and unregistration of the devlink instances under their instance locks. Don't perform the netdev-style wait for all references when unregistering the instance.

Could you explain the reasoning/benefits here? I'm sure some of this is explained in each commit message as well but it would help to understand the overall series.

Fair point, I'll add this:

Yang Yingliang reported [1] a use-after-free in devlink because netdev paths are able to acquire a reference on the devlink instances before they are registered.

[1]
https://lore.kernel.org/all/[email protected]/

#49 lore raw
Quoted 12 lines · Jakub Kicinski
Quoted 10 lines · On Mon, 19 Dec 2022 09:38:09 -0800 Jacob Keller wrote:
Quoted 5 lines · Jakub Kicinski

This set is on top of the previous RFC.

Move the registration and unregistration of the devlink instances under their instance locks. Don't perform the netdev-style wait for all references when unregistering the instance.

Could you explain the reasoning/benefits here? I'm sure some of this is explained in each commit message as well but it would help to understand the overall series.

Quoted 8 lines

Fair point, I'll add this:

Yang Yingliang reported [1] a use-after-free in devlink because netdev paths are able to acquire a reference on the devlink instances before they are registered.

[1]
https://lore.kernel.org/all/[email protected]/

Great. I also think its better to allow some of the sub objects to be setup either before or after registering, as some things might be dynamic. This series lets us get there safely which is good.