Synchronize DT overlay removal with devlink removals

PATCH Applied Herve Codina 4 participants 10 messages Mar 25, 2024 – Mar 26, 2024

Subsystems (3)

Stable branch Driver core Device tree

lore

Share & tools

lore

Herve Codina

author

2024-03-25 15:21 UTC #0 lore raw

v6 cover Synchronize DT overlay removal with devlink removals

Hi,

In the following sequence:
of_platform_depopulate(); /* Remove devices from a DT overlay node */ of_overlay_remove(); /* Remove the DT overlay node itself */

Some warnings are raised by __of_changeset_entry_destroy() which was called from of_overlay_remove():
ERROR: memory leak, expected refcount 1 instead of 2 ...

The issue is that, during the device devlink removals triggered from the of_platform_depopulate(), jobs are put in a workqueue.
These jobs drop the reference to the devices. When a device is no more referenced (refcount == 0), it is released and the reference to its of_node is dropped by a call to of_node_put().
These operations are fully correct except that, because of the workqueue, they are done asynchronously with respect to function calls.

In the sequence provided, the jobs are run too late, after the call to __of_changeset_entry_destroy() and so a missing of_node_put() call is detected by __of_changeset_entry_destroy().

This series fixes this issue introducing device_link_wait_removal() in order to wait for the end of jobs execution (patch 1) and using this function to synchronize the overlay removal with the end of jobs execution (patch 2).

Compared to the previous iteration:
https://lore.kernel.org/linux-kernel/[email protected]/
this v6 series:
- Add Saravana's 'Reviewed-by' tag

This series handles cases reported by Luca [1] and Nuno [2].
[1]: https://lore.kernel.org/all/20231220181627.341e8789@booty/
[2]: https://lore.kernel.org/all/[email protected]/

Best regards,
Hervé

Changes v5 -> v6
- Patch 1
Add 'Reviewed-by: Saravana Kannan <[email protected]>'

- Patch 2
No changes

Changes v4 -> v5
- Patch 1
Remove the 'Fixes' tag
Add 'Tested-by: Luca Ceresoli <[email protected]>' Add 'Reviewed-by: Nuno Sa <[email protected]>'

- Patch 2
Update comment as suggested
Add 'Reviewed-by: Saravana Kannan <[email protected]>' Add 'Tested-by: Luca Ceresoli <[email protected]>' Add 'Reviewed-by: Nuno Sa <[email protected]>'

Changes v3 -> v4
- Patch 1
Uses flush_workqueue() instead of drain_workqueue().

- Patch 2
Remove unlock/re-lock when calling device_link_wait_removal() Move device_link_wait_removal() call to of_changeset_destroy() Update commit log

Changes v2 -> v3
- Patch 1
No changes

- Patch 2
Add missing device.h

Changes v1 -> v2
- Patch 1
Rename the workqueue to 'device_link_wq'
Add 'Fixes' tag and Cc stable

- Patch 2
Add device.h inclusion.
Call device_link_wait_removal() later in the overlay removal sequence (i.e. in free_overlay_changeset() function). Drop of_mutex lock while calling device_link_wait_removal().
Add 'Fixes' tag and Cc stable

Herve Codina (2):
driver core: Introduce device_link_wait_removal()
of: dynamic: Synchronize of_changeset_destroy() with the devlink
removals

 drivers/base/core.c    | 26 +++++++++++++++++++++++---
 drivers/of/dynamic.c   | 12 ++++++++++++
 include/linux/device.h |  1 +
 3 files changed, 36 insertions(+), 3 deletions(-)

Signature · 1 line

2.44.0

Herve Codina

author

2024-03-25 15:21 UTC #1 lore raw

PATCH v6 1/2 driver core: Introduce device_link_wait_removal()

The commit 80dd33cf72d1 ("drivers: base: Fix device link removal") introduces a workqueue to release the consumer and supplier devices used in the devlink.
In the job queued, devices are release and in turn, when all the references to these devices are dropped, the release function of the device itself is called.

Nothing is present to provide some synchronisation with this workqueue in order to ensure that all ongoing releasing operations are done and so, some other operations can be started safely.

For instance, in the following sequence:
1) of_platform_depopulate()
2) of_overlay_remove()

During the step 1, devices are released and related devlinks are removed (jobs pushed in the workqueue).
During the step 2, OF nodes are destroyed but, without any synchronisation with devlink removal jobs, of_overlay_remove() can raise warnings related to missing of_node_put():
ERROR: memory leak, expected refcount 1 instead of 2

Indeed, the missing of_node_put() call is going to be done, too late, from the workqueue job execution.

Introduce device_link_wait_removal() to offer a way to synchronize operations waiting for the end of devlink removals (i.e. end of workqueue jobs).
Also, as a flushing operation is done on the workqueue, the workqueue used is moved from a system-wide workqueue to a local one.

Cc [email protected]

Signed-off-by Herve Codina <[email protected]>

Tested-by Luca Ceresoli <[email protected]>

Reviewed-by Nuno Sa <[email protected]>

Reviewed-by Saravana Kannan <[email protected]>

---
 drivers/base/core.c    | 26 +++++++++++++++++++++++---
 include/linux/device.h |  1 +
 2 files changed, 24 insertions(+), 3 deletions(-)

drivers/base/core.c +23 -3

index 7e3af0ad770a..f2242aadffb0 100644
@@ -44,6 +44,7 @@ static bool fw_devlink_is_permissive(void);
 static void __fw_devlink_link_to_consumers(struct device *dev);
 static bool fw_devlink_drv_reg_done;
 static bool fw_devlink_best_effort;
+static struct workqueue_struct *device_link_wq;
 
 /**
  * __fwnode_link_add - Create a link between two fwnode_handles.

@@ -533,12 +534,26 @@ static void devlink_dev_release(struct device *dev)
 	/*
 	 * It may take a while to complete this work because of the SRCU
 	 * synchronization in device_link_release_fn() and if the consumer or
-	 * supplier devices get deleted when it runs, so put it into the "long"
-	 * workqueue.
+	 * supplier devices get deleted when it runs, so put it into the
+	 * dedicated workqueue.
 	 */
-	queue_work(system_long_wq, &link->rm_work);
+	queue_work(device_link_wq, &link->rm_work);
 }
 
+/**
+ * device_link_wait_removal - Wait for ongoing devlink removal jobs to terminate
+ */
+void device_link_wait_removal(void)
+{
+	/*
+	 * devlink removal jobs are queued in the dedicated work queue.
+	 * To be sure that all removal jobs are terminated, ensure that any
+	 * scheduled work has run to completion.
+	 */
+	flush_workqueue(device_link_wq);
+}
+EXPORT_SYMBOL_GPL(device_link_wait_removal);
+
 static struct class devlink_class = {
 	.name = "devlink",
 	.dev_groups = devlink_groups,

@@ -4165,9 +4180,14 @@ int __init devices_init(void)
 	sysfs_dev_char_kobj = kobject_create_and_add("char", dev_kobj);
 	if (!sysfs_dev_char_kobj)
 		goto char_kobj_err;
+	device_link_wq = alloc_workqueue("device_link_wq", 0, 0);
+	if (!device_link_wq)
+		goto wq_err;
 
 	return 0;
 
+ wq_err:
+	kobject_put(sysfs_dev_char_kobj);
  char_kobj_err:
 	kobject_put(sysfs_dev_block_kobj);
  block_kobj_err:

include/linux/device.h +1 -0

index 1795121dee9a..d7d8305a72e8 100644
@@ -1249,6 +1249,7 @@ void device_link_del(struct device_link *link);
 void device_link_remove(void *consumer, struct device *supplier);
 void device_links_supplier_sync_state_pause(void);
 void device_links_supplier_sync_state_resume(void);
+void device_link_wait_removal(void);
 
 /* Create alias, so I can be autoloaded. */
 #define MODULE_ALIAS_CHARDEV(major,minor) \

Signature · 1 line

2.44.0

Rob Herring

maintainer

2024-03-25 16:44 UTC #2 lore raw

Re: [PATCH v6 1/2] driver core: Introduce device_link_wait_removal()

Quoted 36 lines · Herve Codina

The commit 80dd33cf72d1 ("drivers: base: Fix device link removal") introduces a workqueue to release the consumer and supplier devices used in the devlink.
In the job queued, devices are release and in turn, when all the references to these devices are dropped, the release function of the device itself is called.
Nothing is present to provide some synchronisation with this workqueue in order to ensure that all ongoing releasing operations are done and so, some other operations can be started safely.
For instance, in the following sequence:
1) of_platform_depopulate()
2) of_overlay_remove()
During the step 1, devices are released and related devlinks are removed (jobs pushed in the workqueue).
During the step 2, OF nodes are destroyed but, without any synchronisation with devlink removal jobs, of_overlay_remove() can raise warnings related to missing of_node_put():
ERROR: memory leak, expected refcount 1 instead of 2
Indeed, the missing of_node_put() call is going to be done, too late, from the workqueue job execution.
Introduce device_link_wait_removal() to offer a way to synchronize operations waiting for the end of devlink removals (i.e. end of workqueue jobs).
Also, as a flushing operation is done on the workqueue, the workqueue used is moved from a system-wide workqueue to a local one.
Cc [email protected]
Signed-off-by Herve Codina <[email protected]>
Tested-by Luca Ceresoli <[email protected]>
Reviewed-by Nuno Sa <[email protected]>
Reviewed-by Saravana Kannan <[email protected]>

Quoted 4 lines

---
 drivers/base/core.c    | 26 +++++++++++++++++++++++---
 include/linux/device.h |  1 +
 2 files changed, 24 insertions(+), 3 deletions(-)

Greg, can you ack and I'll take this series.

Rob

Greg Kroah-Hartman

maintainer Acked-by

2024-03-26 08:20 UTC #3 lore raw

Re: [PATCH v6 1/2] driver core: Introduce device_link_wait_removal()

Quoted 42 lines · Rob Herring

Quoted 36 lines · Herve Codina
The commit 80dd33cf72d1 ("drivers: base: Fix device link removal") introduces a workqueue to release the consumer and supplier devices used in the devlink.
In the job queued, devices are release and in turn, when all the references to these devices are dropped, the release function of the device itself is called.
Nothing is present to provide some synchronisation with this workqueue in order to ensure that all ongoing releasing operations are done and so, some other operations can be started safely.
For instance, in the following sequence:
1) of_platform_depopulate()
2) of_overlay_remove()
During the step 1, devices are released and related devlinks are removed (jobs pushed in the workqueue).
During the step 2, OF nodes are destroyed but, without any synchronisation with devlink removal jobs, of_overlay_remove() can raise warnings related to missing of_node_put():
ERROR: memory leak, expected refcount 1 instead of 2
Indeed, the missing of_node_put() call is going to be done, too late, from the workqueue job execution.
Introduce device_link_wait_removal() to offer a way to synchronize operations waiting for the end of devlink removals (i.e. end of workqueue jobs).
Also, as a flushing operation is done on the workqueue, the workqueue used is moved from a system-wide workqueue to a local one.
Cc [email protected]
Signed-off-by Herve Codina <[email protected]>
Tested-by Luca Ceresoli <[email protected]>
Reviewed-by Nuno Sa <[email protected]>
Reviewed-by Saravana Kannan <[email protected]>
Quoted 4 lines
---
 drivers/base/core.c    | 26 +++++++++++++++++++++++---
 include/linux/device.h |  1 +
 2 files changed, 24 insertions(+), 3 deletions(-)

Quoted 1 line

Greg, can you ack and I'll take this series.

Looks semi-sane:

Acked-by Greg Kroah-Hartman <[email protected]>

Greg Kroah-Hartman

maintainer

2024-03-25 18:37 UTC #4 lore raw

Re: [PATCH v6 1/2] driver core: Introduce device_link_wait_removal()

In reply to Herve Codina

Quoted 25 lines · Herve Codina

The commit 80dd33cf72d1 ("drivers: base: Fix device link removal") introduces a workqueue to release the consumer and supplier devices used in the devlink.
In the job queued, devices are release and in turn, when all the references to these devices are dropped, the release function of the device itself is called.
Nothing is present to provide some synchronisation with this workqueue in order to ensure that all ongoing releasing operations are done and so, some other operations can be started safely.
For instance, in the following sequence:
1) of_platform_depopulate()
2) of_overlay_remove()
During the step 1, devices are released and related devlinks are removed (jobs pushed in the workqueue).
During the step 2, OF nodes are destroyed but, without any synchronisation with devlink removal jobs, of_overlay_remove() can raise warnings related to missing of_node_put():
ERROR: memory leak, expected refcount 1 instead of 2
Indeed, the missing of_node_put() call is going to be done, too late, from the workqueue job execution.

Quoted 7 lines

Introduce device_link_wait_removal() to offer a way to synchronize operations waiting for the end of devlink removals (i.e. end of workqueue jobs).
Also, as a flushing operation is done on the workqueue, the workqueue used is moved from a system-wide workqueue to a local one.
Cc [email protected]

Why is this for stable? You are just adding a new api, no one is using it.

Or if they are, you didn't send me that patch...

greg k-h

Herve Codina

author

2024-03-26 07:56 UTC #5 lore raw

Re: [PATCH v6 1/2] driver core: Introduce device_link_wait_removal()

Quoted 32 lines · On Mon, 25 Mar 2024 19:37:10 +0100 Greg Kroah-Hartman…

Quoted 31 lines · Herve Codina
The commit 80dd33cf72d1 ("drivers: base: Fix device link removal") introduces a workqueue to release the consumer and supplier devices used in the devlink.
In the job queued, devices are release and in turn, when all the references to these devices are dropped, the release function of the device itself is called.
Nothing is present to provide some synchronisation with this workqueue in order to ensure that all ongoing releasing operations are done and so, some other operations can be started safely.
For instance, in the following sequence:
1) of_platform_depopulate()
2) of_overlay_remove()
During the step 1, devices are released and related devlinks are removed (jobs pushed in the workqueue).
During the step 2, OF nodes are destroyed but, without any synchronisation with devlink removal jobs, of_overlay_remove() can raise warnings related to missing of_node_put():
ERROR: memory leak, expected refcount 1 instead of 2
Indeed, the missing of_node_put() call is going to be done, too late, from the workqueue job execution.
Introduce device_link_wait_removal() to offer a way to synchronize operations waiting for the end of devlink removals (i.e. end of workqueue jobs).
Also, as a flushing operation is done on the workqueue, the workqueue used is moved from a system-wide workqueue to a local one.

Quoted 6 lines

Quoted 1 line
Cc [email protected]
Why is this for stable? You are just adding a new api, no one is using it.
Or if they are, you didn't send me that patch...

The patch 2 in this current series uses the new api.

Best regards,
Hervé

Greg Kroah-Hartman

maintainer

2024-03-25 18:38 UTC #6 lore raw

Re: [PATCH v6 1/2] driver core: Introduce device_link_wait_removal()

In reply to Herve Codina

Quoted 7 lines · Herve Codina

The commit 80dd33cf72d1 ("drivers: base: Fix device link removal") introduces a workqueue to release the consumer and supplier devices used in the devlink.
In the job queued, devices are release and in turn, when all the references to these devices are dropped, the release function of the device itself is called.

Quoted 7 lines

Nothing is present to provide some synchronisation with this workqueue in order to ensure that all ongoing releasing operations are done and so, some other operations can be started safely.
For instance, in the following sequence:
1) of_platform_depopulate()
2) of_overlay_remove()

So this is only an issue for overlays? Why has no one noticed this in the years since 80dd33cf72d1 was added? Why is this an issue now suddenly?

thanks,

greg k-h

Nuno Sá

2024-03-26 07:05 UTC #7 lore raw

Re: [PATCH v6 1/2] driver core: Introduce device_link_wait_removal()

Quoted 12 lines · Greg Kroah-Hartman

Quoted 11 lines · Herve Codina
The commit 80dd33cf72d1 ("drivers: base: Fix device link removal") introduces a workqueue to release the consumer and supplier devices used in the devlink.
In the job queued, devices are release and in turn, when all the references to these devices are dropped, the release function of the device itself is called.
Nothing is present to provide some synchronisation with this workqueue in order to ensure that all ongoing releasing operations are done and so, some other operations can be started safely.

Quoted 8 lines

Quoted 3 lines
For instance, in the following sequence:
1) of_platform_depopulate()
2) of_overlay_remove()
So this is only an issue for overlays? Why has no one noticed this in the years since 80dd33cf72d1 was added? Why is this an issue now suddenly?

Not sure either... Note this is only an issue if device links are in place. So the overlay needs to have nodes creating those links. You need to have regulators, pwm, eth phy (at least these ones I'm aware they create links) to trigger this. We would have to dig through git to understand when would this be noticeable. But note this is very straight to trigger.

May also very well be that most people don't really "play" with overlay removal... For example, I have been dealing with overlays on rpi's for the last 5 years and only noticed this last year when we had an usecase that involved overlay removal.

- Nuno Sá

Herve Codina

author

2024-03-25 15:21 UTC #8 lore raw

PATCH v6 2/2 of: dynamic: Synchronize of_changeset_destroy() with the devlink removals

In reply to Herve Codina

In the following sequence:
1) of_platform_depopulate()
2) of_overlay_remove()

During the step 1, devices are destroyed and devlinks are removed. During the step 2, OF nodes are destroyed but __of_changeset_entry_destroy() can raise warnings related to missing of_node_put():
ERROR: memory leak, expected refcount 1 instead of 2 ...

Indeed, during the devlink removals performed at step 1, the removal itself releasing the device (and the attached of_node) is done by a job queued in a workqueue and so, it is done asynchronously with respect to function calls.
When the warning is present, of_node_put() will be called but wrongly too late from the workqueue job.

In order to be sure that any ongoing devlink removals are done before the of_node destruction, synchronize the of_changeset_destroy() with the devlink removals.

Fixes 80dd33cf72d1 ("drivers: base: Fix device link removal")

Cc [email protected]

Signed-off-by Herve Codina <[email protected]>

Reviewed-by Saravana Kannan <[email protected]>

Tested-by Luca Ceresoli <[email protected]>

Reviewed-by Nuno Sa <[email protected]>

---
 drivers/of/dynamic.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)

drivers/of/dynamic.c +12 -0

index 3bf27052832f..4d57a4e34105 100644
@@ -9,6 +9,7 @@
 
 #define pr_fmt(fmt)	"OF: " fmt
 
+#include <linux/device.h>
 #include <linux/of.h>
 #include <linux/spinlock.h>
 #include <linux/slab.h>

@@ -667,6 +668,17 @@ void of_changeset_destroy(struct of_changeset *ocs)
 {
 	struct of_changeset_entry *ce, *cen;
 
+	/*
+	 * When a device is deleted, the device links to/from it are also queued
+	 * for deletion. Until these device links are freed, the devices
+	 * themselves aren't freed. If the device being deleted is due to an
+	 * overlay change, this device might be holding a reference to a device
+	 * node that will be freed. So, wait until all already pending device
+	 * links are deleted before freeing a device node. This ensures we don't
+	 * free any device node that has a non-zero reference count.
+	 */
+	device_link_wait_removal();
+
 	list_for_each_entry_safe_reverse(ce, cen, &ocs->entries, node)
 		__of_changeset_entry_destroy(ce);
 }

Signature · 1 line

2.44.0

Rob Herring

maintainer Applied

2024-03-26 20:43 UTC #9 lore raw

In reply to Herve Codina

Quoted 91 lines · Herve Codina

Hi,
In the following sequence:
of_platform_depopulate(); /* Remove devices from a DT overlay node */ of_overlay_remove(); /* Remove the DT overlay node itself */
Some warnings are raised by __of_changeset_entry_destroy() which was called from of_overlay_remove():
ERROR: memory leak, expected refcount 1 instead of 2 ...
The issue is that, during the device devlink removals triggered from the of_platform_depopulate(), jobs are put in a workqueue.
These jobs drop the reference to the devices. When a device is no more referenced (refcount == 0), it is released and the reference to its of_node is dropped by a call to of_node_put().
These operations are fully correct except that, because of the workqueue, they are done asynchronously with respect to function calls.
In the sequence provided, the jobs are run too late, after the call to __of_changeset_entry_destroy() and so a missing of_node_put() call is detected by __of_changeset_entry_destroy().
This series fixes this issue introducing device_link_wait_removal() in order to wait for the end of jobs execution (patch 1) and using this function to synchronize the overlay removal with the end of jobs execution (patch 2).
Compared to the previous iteration:
https://lore.kernel.org/linux-kernel/[email protected]/
this v6 series:
- Add Saravana's 'Reviewed-by' tag
This series handles cases reported by Luca [1] and Nuno [2].
[1]: https://lore.kernel.org/all/20231220181627.341e8789@booty/
[2]: https://lore.kernel.org/all/[email protected]/
Best regards,
Hervé
Changes v5 -> v6
- Patch 1
Add 'Reviewed-by: Saravana Kannan <[email protected]>'
- Patch 2
No changes
Changes v4 -> v5
- Patch 1
Remove the 'Fixes' tag
Add 'Tested-by: Luca Ceresoli <[email protected]>' Add 'Reviewed-by: Nuno Sa <[email protected]>'
- Patch 2
Update comment as suggested
Add 'Reviewed-by: Saravana Kannan <[email protected]>' Add 'Tested-by: Luca Ceresoli <[email protected]>' Add 'Reviewed-by: Nuno Sa <[email protected]>'
Changes v3 -> v4
- Patch 1
Uses flush_workqueue() instead of drain_workqueue().
- Patch 2
Remove unlock/re-lock when calling device_link_wait_removal() Move device_link_wait_removal() call to of_changeset_destroy() Update commit log
Changes v2 -> v3
- Patch 1
No changes
- Patch 2
Add missing device.h
Changes v1 -> v2
- Patch 1
Rename the workqueue to 'device_link_wq'
Add 'Fixes' tag and Cc stable
- Patch 2
Add device.h inclusion.
Call device_link_wait_removal() later in the overlay removal sequence (i.e. in free_overlay_changeset() function). Drop of_mutex lock while calling device_link_wait_removal().
Add 'Fixes' tag and Cc stable
Herve Codina (2):
driver core: Introduce device_link_wait_removal()
of: dynamic: Synchronize of_changeset_destroy() with the devlink
removals

Quoted 4 lines

 drivers/base/core.c    | 26 +++++++++++++++++++++++---
 drivers/of/dynamic.c   | 12 ++++++++++++
 include/linux/device.h |  1 +
 3 files changed, 36 insertions(+), 3 deletions(-)

Applied, thanks!