Skip to contentSkip to conversation

Solve postboot supplier cleanup and optimize probe ordering

PATCH Superseded Saravana Kannan 3 participants 7 messages Jun 28, 2019 – Jul 2, 2019

Subsystems (2)

v2 of 11 Superseded Latest is v11 →

lore
Share & tools
lore
#0 lore raw

v2 cover Solve postboot supplier cleanup and optimize probe ordering

Add device-links to track functional dependencies between devices after they are created (but before they are probed) by looking at their common DT bindings like clocks, interconnects, etc.

Having functional dependencies automatically added before the devices are probed, provides the following benefits:

- Optimizes device probe order and avoids the useless work of attempting probes of devices that will not probe successfully (because their suppliers aren't present or haven't probed yet).

For example, in a commonly available mobile SoC, registering just one consumer device's driver at an initcall level earlier than the supplier device's driver causes 11 failed probe attempts before the consumer device probes successfully. This was with a kernel with all the drivers statically compiled in. This problem gets a lot worse if all the drivers are loaded as modules without direct symbol dependencies.

- Supplier devices like clock providers, interconnect providers, etc need to keep the resources they provide active and at a particular state(s) during boot up even if their current set of consumers don't request the resource to be active. This is because the rest of the consumers might not have probed yet and turning off the resource before all the consumers have probed could lead to a hang or undesired user experience.

Some frameworks (Eg: regulator) handle this today by turning off "unused" resources at late_initcall_sync and hoping all the devices have probed by then. This is not a valid assumption for systems with loadable modules. Other frameworks (Eg: clock) just don't handle this due to the lack of a clear signal for when they can turn off resources. This leads to downstream hacks to handle cases like this that can easily be solved in the upstream kernel.

By linking devices before they are probed, we give suppliers a clear count of the number of dependent consumers. Once all of the consumers are active, the suppliers can turn off the unused resources without making assumptions about the number of consumers.

By default we just add device-links to track "driver presence" (probe succeeded) of the supplier device. If any other functionality provided by device-links are needed, it is left to the consumer/supplier devices to change the link when they probe.

Saravana Kannan (3):
driver core: Add device links support for pending links to suppliers of/platform: Add functional dependency link from DT bindings driver core: Add sync_state driver/bus callback

 drivers/base/core.c    | 106 +++++++++++++++++++++++++++++++++++++++++
 drivers/of/Kconfig     |   9 ++++
 drivers/of/platform.c  |  82 +++++++++++++++++++++++++++++++
 include/linux/device.h |  24 ++++++++++
 4 files changed, 221 insertions(+)
Signature · 5 lines
v1 -> v2
- Drop patch to speed up of_find_device_by_node()
- Drop depends-on property and use existing bindings

2.22.0.410.gd8fdbe21b5-goog
#1 lore raw

PATCH v2 1/3 driver core: Add device links support for pending links to suppliers

When consumer devices are added, they might not have a supplier device to link to despite needing mandatory resources/functionality from one or more suppliers. Add a waiting_for_suppliers list to track such consumers and add helper functions to manage the list.

Marking/unmarking a consumer device as waiting for suppliers is generally expected to be done by the entity that's creating the device.

---
 drivers/base/core.c    | 67 ++++++++++++++++++++++++++++++++++++++++++
 include/linux/device.h |  5 ++++
 2 files changed, 72 insertions(+)
drivers/base/core.c +67 -0
index fd7511e04e62..9ab6782dda1c 100644
@@ -44,6 +44,8 @@ early_param("sysfs.deprecated", sysfs_deprecated_setup);
 #endif
 
 /* Device links support. */
+static LIST_HEAD(wait_for_suppliers);
+static DEFINE_MUTEX(wfs_lock);
 
 #ifdef CONFIG_SRCU
 static DEFINE_MUTEX(device_links_lock);
@@ -401,6 +403,53 @@ struct device_link *device_link_add(struct device *consumer,
 }
 EXPORT_SYMBOL_GPL(device_link_add);
 
+/**
+ * device_link_wait_for_supplier - Mark device as waiting for supplier
+ * @consumer: Consumer device
+ *
+ * Marks the consumer device as waiting for suppliers to become available. The
+ * consumer device will never be probed until it's unmarked as waiting for
+ * suppliers. The caller is responsible for adding the link to the supplier
+ * once the supplier device is present.
+ *
+ * This function is NOT meant to be called from the probe function of the
+ * consumer but rather from code that creates the consumer device.
+ */
+void device_link_wait_for_supplier(struct device *consumer)
+{
+	mutex_lock(&wfs_lock);
+	list_add_tail(&consumer->links.needs_suppliers, &wait_for_suppliers);
+	mutex_unlock(&wfs_lock);
+}
+
+/**
+ * device_link_check_waiting_consumers - Try to unmark waiting consumers
+ * @add_suppliers: Callback function to add suppliers to waiting consumer
+ *
+ * Loops through all consumers waiting on suppliers and tries to add all their
+ * supplier links. If that succeeds, the consumer device is unmarked as waiting
+ * for suppliers. Otherwise, they are left marked as waiting on suppliers,
+ *
+ * The add_suppliers callback is expected to return 0 if it has found and added
+ * all the supplier links for the consumer device. It should return an error if
+ * it isn't able to do so.
+ *
+ * The caller of device_link_wait_for_supplier() is expected to call this once
+ * it's aware of potential suppliers becoming available.
+ */
+void device_link_check_waiting_consumers(
+		int (*add_suppliers)(struct device *consumer))
+{
+	struct device *dev, *tmp;
+
+	mutex_lock(&wfs_lock);
+	list_for_each_entry_safe(dev, tmp, &wait_for_suppliers,
+				 links.needs_suppliers)
+		if (!add_suppliers(dev))
+			list_del_init(&dev->links.needs_suppliers);
+	mutex_unlock(&wfs_lock);
+}
+
 static void device_link_free(struct device_link *link)
 {
 	while (refcount_dec_not_one(&link->rpm_active))
@@ -535,6 +584,19 @@ int device_links_check_suppliers(struct device *dev)
 	struct device_link *link;
 	int ret = 0;
 
+	/*
+	 * If a device is waiting for one or more suppliers (in
+	 * wait_for_suppliers list), it is not ready to probe yet. So just
+	 * return -EPROBE_DEFER without having to check the links with existing
+	 * suppliers.
+	 */
+	mutex_lock(&wfs_lock);
+	if (!list_empty(&dev->links.needs_suppliers)) {
+		mutex_unlock(&wfs_lock);
+		return -EPROBE_DEFER;
+	}
+	mutex_unlock(&wfs_lock);
+
 	device_links_write_lock();
 
 	list_for_each_entry(link, &dev->links.suppliers, c_node) {
@@ -812,6 +874,10 @@ static void device_links_purge(struct device *dev)
 {
 	struct device_link *link, *ln;
 
+	mutex_lock(&wfs_lock);
+	list_del(&dev->links.needs_suppliers);
+	mutex_unlock(&wfs_lock);
+
 	/*
 	 * Delete all of the remaining links from this device to any other
 	 * devices (either consumers or suppliers).
@@ -1673,6 +1739,7 @@ void device_initialize(struct device *dev)
 #endif
 	INIT_LIST_HEAD(&dev->links.consumers);
 	INIT_LIST_HEAD(&dev->links.suppliers);
+	INIT_LIST_HEAD(&dev->links.needs_suppliers);
 	dev->links.status = DL_DEV_NO_DRIVER;
 }
 EXPORT_SYMBOL_GPL(device_initialize);
include/linux/device.h +5 -0
index 848fc71c6ba6..026dd842e511 100644
@@ -888,11 +888,13 @@ enum dl_dev_state {
  * struct dev_links_info - Device data related to device links.
  * @suppliers: List of links to supplier devices.
  * @consumers: List of links to consumer devices.
+ * @needs_suppliers: Hook to global list of devices waiting for suppliers.
  * @status: Driver status information.
  */
 struct dev_links_info {
 	struct list_head suppliers;
 	struct list_head consumers;
+	struct list_head needs_suppliers;
 	enum dl_dev_state status;
 };
 
@@ -1396,6 +1398,9 @@ struct device_link *device_link_add(struct device *consumer,
 				    struct device *supplier, u32 flags);
 void device_link_del(struct device_link *link);
 void device_link_remove(void *consumer, struct device *supplier);
+void device_link_wait_for_supplier(struct device *consumer);
+void device_link_check_waiting_consumers(
+		int (*add_suppliers)(struct device *consumer));
 
 #ifndef dev_fmt
 #define dev_fmt(fmt) fmt
Signature · 1 line
2.22.0.410.gd8fdbe21b5-goog
#2 lore raw

PATCH v2 2/3 of/platform: Add functional dependency link from DT bindings

In reply to Saravana Kannan

Add device-links to track functional dependencies between devices after they are created (but before they are probed) by looking at their common DT bindings like clocks, interconnects, etc.

Automatically adding device-links to track functional dependencies at the framework level provides the following benefits:

- Optimizes device probe order and avoids the useless work of attempting probes of devices that will not probe successfully (because their suppliers aren't present or haven't probed yet).

For example, in a commonly available mobile SoC, registering just one consumer device's driver at an initcall level earlier than the supplier device's driver causes 11 failed probe attempts before the consumer device probes successfully. This was with a kernel with all the drivers statically compiled in. This problem gets a lot worse if all the drivers are loaded as modules without direct symbol dependencies.

- Supplier devices like clock providers, interconnect providers, etc need to keep the resources they provide active and at a particular state(s) during boot up even if their current set of consumers don't request the resource to be active. This is because the rest of the consumers might not have probed yet and turning off the resource before all the consumers have probed could lead to a hang or undesired user experience.

Some frameworks (Eg: regulator) handle this today by turning off "unused" resources at late_initcall_sync and hoping all the devices have probed by then. This is not a valid assumption for systems with loadable modules. Other frameworks (Eg: clock) just don't handle this due to the lack of a clear signal for when they can turn off resources. This leads to downstream hacks to handle cases like this that can easily be solved in the upstream kernel.

By linking devices before they are probed, we give suppliers a clear count of the number of dependent consumers. Once all of the consumers are active, the suppliers can turn off the unused resources without making assumptions about the number of consumers.

By default we just add device-links to track "driver presence" (probe succeeded) of the supplier device. If any other functionality provided by device-links are needed, it is left to the consumer/supplier devices to change the link when they probe.

---
 drivers/of/Kconfig    |  9 ++++++
 drivers/of/platform.c | 73 +++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 82 insertions(+)
drivers/of/Kconfig +9 -0
index 37c2ccbefecd..7c7fa7394b4c 100644
@@ -103,4 +103,13 @@ config OF_OVERLAY
 config OF_NUMA
 	bool
 
+config OF_DEVLINKS
+	bool "Device links from DT bindings"
+	help
+	  Common DT bindings like clocks, interconnects, etc represent a
+	  consumer device's dependency on suppliers devices. This option
+	  creates device links from these common bindings so that consumers are
+	  probed only after all their suppliers are active and suppliers can
+	  tell when all their consumers are active.
+
 endif # OF
drivers/of/platform.c +73 -0

95 lines not on this page · raw message

Signature · 1 line
2.22.0.410.gd8fdbe21b5-goog
#3 lore raw

Re: [PATCH v2 2/3] of/platform: Add functional dependency link from DT bindings

Hello Saravana,

Quoted 11 lines · Saravana Kannan
drivers/of/platform.c +3 -0
index 04ad312fd85b..8d690fa0f47c 100644
@@ -61,6 +61,72 @@ struct platform_device *of_find_device_by_node(struct device_node *np)
 EXPORT_SYMBOL(of_find_device_by_node);
 
 #ifdef CONFIG_OF_ADDRESS
+static int of_link_binding(struct device *dev, char *binding, char *cell)
+{
+	struct of_phandle_args sup_args;

drivers/of/platform.c struct platform_device *of_find_device_by_node(struct device_node *np) · line 76

Quoted 10 lines
drivers/of/platform.c +10 -0
+	struct platform_device *sup_dev;
+	unsigned int i = 0, links = 0;
+	u32 dl_flags = DL_FLAG_AUTOPROBE_CONSUMER;
+
+	while (!of_parse_phandle_with_args(dev->of_node, binding, cell, i,
+					   &sup_args)) {
+		i++;
+		sup_dev = of_find_device_by_node(sup_args.np);
+		if (!sup_dev)
+			continue;

This check means that a required dependency link between a consumer and supplier will not be added in the case that the consumer device is created before the supply device. If the supplier device is created and immediately bound to its driver after late_initcall_sync(), then it is possible for the sync_state() callback of the supplier to be called before the consumer gets a chance to probe since its link was never captured.

of_platform_default_populate() below will only create devices for the first level DT nodes directly under "/". Suppliers DT nodes can exist as second level nodes under a first level bus node (e.g. I2C, SPMI, RPMh, etc). Thus, it is quite likely that not all supplier devices will have been created when device_link_check_waiting_consumers() is called.

As far as I can tell, this effectively breaks the sync_state() functionality (and thus proxy un-voting built on top of it) when using kernel modules for both the supplier and consumer drivers which are probed after late_initcall_sync(). I'm not sure how this can be avoided given that the linking is done between devices in the process of sequentially adding devices. Perhaps linking between device nodes instead of devices might be able to overcome this issue.

Quoted 9 lines
drivers/of/platform.c +9 -0
+		if (device_link_add(dev, &sup_dev->dev, dl_flags))
+			links++;
+		put_device(&sup_dev->dev);
+	}
+	if (links < i)
+		return -ENODEV;
+	return 0;
+}
+

drivers/of/platform.c struct platform_device *of_find_device_by_node(struct device_node *np) · line 95

Quoted 10 lines
drivers/of/platform.c +10 -0
+/*
+ * List of bindings and their cell names (use NULL if no cell names) from which
+ * device links need to be created.
+ */
+static char *link_bindings[] = {
+#ifdef CONFIG_OF_DEVLINKS
+	"clocks", "#clock-cells",
+	"interconnects", "#interconnect-cells",
+#endif
+};

This list and helper function above are missing support for regulator <arbitrary-consumer-name>-supply properties. We require this support on QTI boards in order to handle regulator proxy un-voting when booting with kernel modules. Are you planning to add this support in a follow-on version of this patch or in an additional patch?

Note that handling regulator supply properties will be very challenging for at least these reasons:

1. There is not a consistent DT property name used for regulator supplies.

2. The device node referenced in a regulator supply phandle is usually not the device node which correspond to the device pointer for the supplier. This is because a single regulator supplier device node (which will have an associated device pointer) typically has a subnode for each of the regulators it supports. Consumers then use phandles for the subnodes.

3. The specification of parent supplies for regulators frequently results in *-supply properties in a node pointing to child subnodes of that node. See [1] for an example. Special care would need to be taken to avoid
trying to mark a regulator supplier as a supplier to itself as well as to avoid blocking its own probing due to an unlinked supply dependency.

4. Not all DT properties of the form "*-supply" are regulator supplies. (Note, this case has been discussed, but I was not able to locate an example of it.)

Clocks also have a problem. A recent patch [2] allows clock provider parent clocks to be specified via DT. This could lead to cases of circular "clocks" property dependencies where there are two clock supplier devices A and B with A having some clocks with B clock parents along with B having some clocks with A clock parents. If "clocks" properties are followed, then neither device would ever be able to probe.

This does not present a problem without this patch series because the clock framework supports late binding of parents specifically to avoid issues with clocks not registering in perfectly topological order of parent dependencies.

Quoted 53 lines
drivers/of/platform.c +37 -0
+
+static int of_link_to_suppliers(struct device *dev)
+{
+	unsigned int i = 0;
+	bool done = true;
+
+	if (unlikely(!dev->of_node))
+		return 0;
+
+	for (i = 0; i < ARRAY_SIZE(link_bindings) / 2; i++)
+		if (of_link_binding(dev, link_bindings[i * 2],
+					link_bindings[i * 2 + 1]))
+			done = false;
+
+	if (!done)
+		return -ENODEV;
+	return 0;
+}
+
+static void link_waiting_consumers_func(struct work_struct *work)
+{
+	device_link_check_waiting_consumers(of_link_to_suppliers);
+}
+static DECLARE_WORK(link_waiting_consumers_work, link_waiting_consumers_func);
+
+static bool link_waiting_consumers_enable;
+static void link_waiting_consumers_trigger(void)
+{
+	if (!link_waiting_consumers_enable)
+		return;
+
+	schedule_work(&link_waiting_consumers_work);
+}
+
 /*
  * The following routines scan a subtree and registers a device for
  * each applicable node.
@@ -192,10 +258,13 @@ static struct platform_device *of_platform_device_create_pdata(
 	dev->dev.platform_data = platform_data;
 	of_msi_configure(&dev->dev, dev->dev.of_node);
 
+	if (of_link_to_suppliers(&dev->dev))
+		device_link_wait_for_supplier(&dev->dev);
 	if (of_device_add(dev) != 0) {
 		platform_device_put(dev);
 		goto err_clear_flag;
 	}
+	link_waiting_consumers_trigger();
 
 	return dev;
 

drivers/of/platform.c static int __init of_platform_default_populate_init(void) · line 619

Quoted 10 lines
drivers/of/platform.c +4 -0
@@ -541,6 +610,10 @@ static int __init of_platform_default_populate_init(void)
 	/* Populate everything else. */
 	of_platform_default_populate(NULL, NULL, NULL);
 
+	/* Make the device-links between suppliers and consumers */
+	link_waiting_consumers_enable = true;
+	device_link_check_waiting_consumers(of_link_to_suppliers);
+
 	return 0;
 }
 arch_initcall_sync(of_platform_default_populate_init);

Thanks,
David

[1]:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/arm64/boot/dts/qcom/sdm845-mtp.dts?h=v5.2-rc5#n73

[2]:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=fc0c209c147f35ed2648adda09db39fcad89e334

Signature · 2 lines
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project
#4 lore raw

Re: [PATCH v2 2/3] of/platform: Add functional dependency link from DT bindings

Quoted 24 lines · David Collins

Hello Saravana,

Quoted 20 lines · Saravana Kannan
drivers/of/platform.c +12 -0
index 04ad312fd85b..8d690fa0f47c 100644
@@ -61,6 +61,72 @@ struct platform_device *of_find_device_by_node(struct device_node *np)
 EXPORT_SYMBOL(of_find_device_by_node);
 
 #ifdef CONFIG_OF_ADDRESS
+static int of_link_binding(struct device *dev, char *binding, char *cell)
+{
+     struct of_phandle_args sup_args;
+     struct platform_device *sup_dev;
+     unsigned int i = 0, links = 0;
+     u32 dl_flags = DL_FLAG_AUTOPROBE_CONSUMER;
+
+     while (!of_parse_phandle_with_args(dev->of_node, binding, cell, i,
+                                        &sup_args)) {
+             i++;
+             sup_dev = of_find_device_by_node(sup_args.np);
+             if (!sup_dev)
Quoted 8 lines

drivers/of/platform.c struct platform_device *of_find_device_by_node(struct device_node *np) · line 76

Quoted 1 line
drivers/of/platform.c +1 -0
+                     continue;

This check means that a required dependency link between a consumer and supplier will not be added in the case that the consumer device is created before the supply device. If the supplier device is created and immediately bound to its driver after late_initcall_sync(), then it is possible for the sync_state() callback of the supplier to be called before the consumer gets a chance to probe since its link was never captured.

Yeah, I was aware of this but wasn't sure how likely this case was. I didn't want to go down the rabbit hole of handling every corner case perfectly before seeing how the general idea was received by the maintainers. Also, was waiting to see if someone complained about it before trying to fix it.

Quoted 5 lines

of_platform_default_populate() below will only create devices for the first level DT nodes directly under "/". Suppliers DT nodes can exist as second level nodes under a first level bus node (e.g. I2C, SPMI, RPMh, etc). Thus, it is quite likely that not all supplier devices will have been created when device_link_check_waiting_consumers() is called.

Yeah, those are all good example of when this could be an issue.

Quoted 7 lines

As far as I can tell, this effectively breaks the sync_state() functionality (and thus proxy un-voting built on top of it) when using kernel modules for both the supplier and consumer drivers which are probed after late_initcall_sync(). I'm not sure how this can be avoided given that the linking is done between devices in the process of sequentially adding devices. Perhaps linking between device nodes instead of devices might be able to overcome this issue.

I'm not sure linking struct device_node would be useful here. There are different and simpler ways of fixing it. Working on them right now (v3 patch series). Thanks for bringing up the good examples.

Quoted 19 lines
Quoted 19 lines
drivers/of/platform.c +17 -0
+             if (device_link_add(dev, &sup_dev->dev, dl_flags))
+                     links++;
+             put_device(&sup_dev->dev);
+     }
+     if (links < i)
+             return -ENODEV;
+     return 0;
+}
+
+/*
+ * List of bindings and their cell names (use NULL if no cell names) from which
+ * device links need to be created.
+ */
+static char *link_bindings[] = {
+#ifdef CONFIG_OF_DEVLINKS
+     "clocks", "#clock-cells",
+     "interconnects", "#interconnect-cells",
Quoted 8 lines

drivers/of/platform.c struct platform_device *of_find_device_by_node(struct device_node *np) · line 95

Quoted 2 lines
drivers/of/platform.c +2 -0
+#endif
+};

This list and helper function above are missing support for regulator <arbitrary-consumer-name>-supply properties. We require this support on QTI boards in order to handle regulator proxy un-voting when booting with kernel modules. Are you planning to add this support in a follow-on version of this patch or in an additional patch?

Yes, I intentionally left out regulators here because it's a huge can of worms. But keep in mind, that even without adding regulator DT binding handling here, you could still switch to sync_state callback and be no worse than you are today. Once regulator supplier-consumer linking is added/improved, the QTI boards would start working with modules.

As for how regulators supplier-consumer linking is handled, I think that's the one we need to discuss and figure out. But I don't think the regulator binding necessarily has to be handled in this patch series. I'm sure in general the number of bindings we support could be improved over time.

Quoted 5 lines

Note that handling regulator supply properties will be very challenging for at least these reasons:

1. There is not a consistent DT property name used for regulator supplies.

Yup. Maybe we can add a new regulator binding format with a more consistent name (like clocks and interconnects) and deprecate the older ones? Seems like a need binding clean up in general.

Quoted 5 lines

2. The device node referenced in a regulator supply phandle is usually not the device node which correspond to the device pointer for the supplier. This is because a single regulator supplier device node (which will have an associated device pointer) typically has a subnode for each of the regulators it supports. Consumers then use phandles for the subnodes.

If I'm not mistaken, looks like this can be multiple sub-nodes deep too. One option is to walk up the phandle till we find a compatible string and then find the device for that node?

Quoted 5 lines

3. The specification of parent supplies for regulators frequently results in *-supply properties in a node pointing to child subnodes of that node. See [1] for an example. Special care would need to be taken to avoid
trying to mark a regulator supplier as a supplier to itself as well as to avoid blocking its own probing due to an unlinked supply dependency.

Sigh... as if it's not already complicated enough. Anyway, device_link_add() already has a bunch of check to avoid creating cyclic dependencies, etc. So, I'd expect this to be handled already. At worst case, we might need to add a few more checks there. But that hopefully shouldn't be an issue.

Quoted 3 lines

4. Not all DT properties of the form "*-supply" are regulator supplies. (Note, this case has been discussed, but I was not able to locate an example of it.)

Yup and I hate this part. Not sure what to say.

Quoted 6 lines

Clocks also have a problem. A recent patch [2] allows clock provider parent clocks to be specified via DT. This could lead to cases of circular "clocks" property dependencies where there are two clock supplier devices A and B with A having some clocks with B clock parents along with B having some clocks with A clock parents. If "clocks" properties are followed, then neither device would ever be able to probe.

Interconnects have a similar problem too because every interconnect lists all the other interconnects it's connected to. Even if that's magically addressed correctly, interconnect consumers still have a problem because "interconnect" DT binding only lists phandles of the source and destination interconnect and not all the interconnect along the way. So they will be missing dependencies.

In general I agree with your points about clocks. I've brought this up multiple times, but the maintainers insists I first implement parsing existing DT bindings. So, I've done that. Lets see what they have to say now.

But I have a few more ideas for handling circular dependencies without adding new DT bindings that might work (will send out as part of v3 patch series) but interconnects are still an issue.

Quoted 4 lines

This does not present a problem without this patch series because the clock framework supports late binding of parents specifically to avoid issues with clocks not registering in perfectly topological order of parent dependencies.

That's why I added the OF_DEVLINKS config. As of v2, you simply can't use it for SoC/boards with cyclic clock dependencies. But again, sync_state is no worse that what's there today. And it'll only improve over time.

-Saravana

Quoted 6 lines

[2]:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=fc0c209c147f35ed2648adda09db39fcad89e334

--
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project

#5 lore raw

Re: [PATCH v2 2/3] of/platform: Add functional dependency link from DT bindings

In reply to David Collins
Quoted 91 lines · David Collins

Hello Saravana,

Quoted 11 lines · Saravana Kannan
drivers/of/platform.c +3 -0
index 04ad312fd85b..8d690fa0f47c 100644
@@ -61,6 +61,72 @@ struct platform_device *of_find_device_by_node(struct device_node *np)
 EXPORT_SYMBOL(of_find_device_by_node);
 
 #ifdef CONFIG_OF_ADDRESS
+static int of_link_binding(struct device *dev, char *binding, char *cell)
+{
+     struct of_phandle_args sup_args;

drivers/of/platform.c struct platform_device *of_find_device_by_node(struct device_node *np) · line 76

Quoted 10 lines
drivers/of/platform.c +10 -0
+     struct platform_device *sup_dev;
+     unsigned int i = 0, links = 0;
+     u32 dl_flags = DL_FLAG_AUTOPROBE_CONSUMER;
+
+     while (!of_parse_phandle_with_args(dev->of_node, binding, cell, i,
+                                        &sup_args)) {
+             i++;
+             sup_dev = of_find_device_by_node(sup_args.np);
+             if (!sup_dev)
+                     continue;

This check means that a required dependency link between a consumer and supplier will not be added in the case that the consumer device is created before the supply device. If the supplier device is created and immediately bound to its driver after late_initcall_sync(), then it is possible for the sync_state() callback of the supplier to be called before the consumer gets a chance to probe since its link was never captured.

of_platform_default_populate() below will only create devices for the first level DT nodes directly under "/". Suppliers DT nodes can exist as second level nodes under a first level bus node (e.g. I2C, SPMI, RPMh, etc). Thus, it is quite likely that not all supplier devices will have been created when device_link_check_waiting_consumers() is called.

As far as I can tell, this effectively breaks the sync_state() functionality (and thus proxy un-voting built on top of it) when using kernel modules for both the supplier and consumer drivers which are probed after late_initcall_sync(). I'm not sure how this can be avoided given that the linking is done between devices in the process of sequentially adding devices. Perhaps linking between device nodes instead of devices might be able to overcome this issue.

Quoted 9 lines
drivers/of/platform.c +9 -0
+             if (device_link_add(dev, &sup_dev->dev, dl_flags))
+                     links++;
+             put_device(&sup_dev->dev);
+     }
+     if (links < i)
+             return -ENODEV;
+     return 0;
+}
+

drivers/of/platform.c struct platform_device *of_find_device_by_node(struct device_node *np) · line 95

Quoted 10 lines
drivers/of/platform.c +10 -0
+/*
+ * List of bindings and their cell names (use NULL if no cell names) from which
+ * device links need to be created.
+ */
+static char *link_bindings[] = {
+#ifdef CONFIG_OF_DEVLINKS
+     "clocks", "#clock-cells",
+     "interconnects", "#interconnect-cells",
+#endif
+};

This list and helper function above are missing support for regulator <arbitrary-consumer-name>-supply properties. We require this support on QTI boards in order to handle regulator proxy un-voting when booting with kernel modules. Are you planning to add this support in a follow-on version of this patch or in an additional patch?

Note that handling regulator supply properties will be very challenging for at least these reasons:

1. There is not a consistent DT property name used for regulator supplies.

2. The device node referenced in a regulator supply phandle is usually not the device node which correspond to the device pointer for the supplier. This is because a single regulator supplier device node (which will have an associated device pointer) typically has a subnode for each of the regulators it supports. Consumers then use phandles for the subnodes.

3. The specification of parent supplies for regulators frequently results in *-supply properties in a node pointing to child subnodes of that node. See [1] for an example. Special care would need to be taken to avoid
trying to mark a regulator supplier as a supplier to itself as well as to avoid blocking its own probing due to an unlinked supply dependency.

Quoted 3 lines

4. Not all DT properties of the form "*-supply" are regulator supplies. (Note, this case has been discussed, but I was not able to locate an example of it.)

I don't think this is true. There's a

Quoted 91 lines

Clocks also have a problem. A recent patch [2] allows clock provider parent clocks to be specified via DT. This could lead to cases of circular "clocks" property dependencies where there are two clock supplier devices A and B with A having some clocks with B clock parents along with B having some clocks with A clock parents. If "clocks" properties are followed, then neither device would ever be able to probe.

This does not present a problem without this patch series because the clock framework supports late binding of parents specifically to avoid issues with clocks not registering in perfectly topological order of parent dependencies.

Quoted 53 lines
drivers/of/platform.c +37 -0
+
+static int of_link_to_suppliers(struct device *dev)
+{
+     unsigned int i = 0;
+     bool done = true;
+
+     if (unlikely(!dev->of_node))
+             return 0;
+
+     for (i = 0; i < ARRAY_SIZE(link_bindings) / 2; i++)
+             if (of_link_binding(dev, link_bindings[i * 2],
+                                     link_bindings[i * 2 + 1]))
+                     done = false;
+
+     if (!done)
+             return -ENODEV;
+     return 0;
+}
+
+static void link_waiting_consumers_func(struct work_struct *work)
+{
+     device_link_check_waiting_consumers(of_link_to_suppliers);
+}
+static DECLARE_WORK(link_waiting_consumers_work, link_waiting_consumers_func);
+
+static bool link_waiting_consumers_enable;
+static void link_waiting_consumers_trigger(void)
+{
+     if (!link_waiting_consumers_enable)
+             return;
+
+     schedule_work(&link_waiting_consumers_work);
+}
+
 /*
  * The following routines scan a subtree and registers a device for
  * each applicable node.
@@ -192,10 +258,13 @@ static struct platform_device *of_platform_device_create_pdata(
      dev->dev.platform_data = platform_data;
      of_msi_configure(&dev->dev, dev->dev.of_node);
 
+     if (of_link_to_suppliers(&dev->dev))
+             device_link_wait_for_supplier(&dev->dev);
      if (of_device_add(dev) != 0) {
              platform_device_put(dev);
              goto err_clear_flag;
      }
+     link_waiting_consumers_trigger();
 
      return dev;
 

drivers/of/platform.c static int __init of_platform_default_populate_init(void) · line 619

Quoted 10 lines
drivers/of/platform.c +4 -0
@@ -541,6 +610,10 @@ static int __init of_platform_default_populate_init(void)
      /* Populate everything else. */
      of_platform_default_populate(NULL, NULL, NULL);
 
+     /* Make the device-links between suppliers and consumers */
+     link_waiting_consumers_enable = true;
+     device_link_check_waiting_consumers(of_link_to_suppliers);
+
      return 0;
 }
 arch_initcall_sync(of_platform_default_populate_init);

Thanks,
David

[1]:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/arm64/boot/dts/qcom/sdm845-mtp.dts?h=v5.2-rc5#n73

[2]:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=fc0c209c147f35ed2648adda09db39fcad89e334

--
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project

#6 lore raw

PATCH v2 3/3 driver core: Add sync_state driver/bus callback

In reply to Saravana Kannan

This sync_state driver/bus callback is called once all the consumers of a supplier have probed successfully.

This allows the supplier device's driver/bus to sync the supplier device's state to the software state with the guarantee that all the consumers are actively managing the resources provided by the supplier device.

To maintain backwards compatibility and ease transition from existing frameworks and resource cleanup schemes, late_initcall_sync is the earliest when the sync_state callback might be called.

There is no upper bound on the time by which the sync_state callback has to be called. This is because if a consumer device never probes, the supplier has to maintain its resources in the state left by the bootloader. For example, if the bootloader leaves the display backlight at a fixed voltage and the backlight driver is never probed, you don't want the backlight to ever be turned off after boot up.

---
 drivers/base/core.c    | 39 +++++++++++++++++++++++++++++++++++++++
 drivers/of/platform.c  |  9 +++++++++
 include/linux/device.h | 19 +++++++++++++++++++
 3 files changed, 67 insertions(+)
drivers/base/core.c +39 -0
index 9ab6782dda1c..7a8777a33e8c 100644
@@ -46,6 +46,7 @@ early_param("sysfs.deprecated", sysfs_deprecated_setup);
 /* Device links support. */
 static LIST_HEAD(wait_for_suppliers);
 static DEFINE_MUTEX(wfs_lock);
+static bool supplier_sync_state_enabled;
 
 #ifdef CONFIG_SRCU
 static DEFINE_MUTEX(device_links_lock);
@@ -616,6 +617,41 @@ int device_links_check_suppliers(struct device *dev)
 	return ret;
 }
 
+static void __device_links_supplier_sync_state(struct device *dev)
+{
+	struct device_link *link;
+
+	if (dev->state_synced)
+		return;
+
+	list_for_each_entry(link, &dev->links.consumers, s_node) {
+		if (link->flags & DL_FLAG_STATELESS)
+			continue;
+		if (link->status != DL_STATE_ACTIVE)
+			return;
+	}
+
+	if (dev->bus->sync_state)
+		dev->bus->sync_state(dev);
+	else if (dev->driver && dev->driver->sync_state)
+		dev->driver->sync_state(dev);
+
+	dev->state_synced = true;
+}
+
+int device_links_supplier_sync_state(struct device *dev, void *data)
+{
+	device_links_write_lock();
+	__device_links_supplier_sync_state(dev);
+	device_links_write_unlock();
+	return 0;
+}
+
+void device_links_supplier_sync_state_enable(void)
+{
+	supplier_sync_state_enabled = true;
+}
+
 /**
  * device_links_driver_bound - Update device links after probing its driver.
  * @dev: Device to update the links for.
@@ -660,6 +696,9 @@ void device_links_driver_bound(struct device *dev)
 
 		WARN_ON(link->status != DL_STATE_CONSUMER_PROBE);
 		WRITE_ONCE(link->status, DL_STATE_ACTIVE);
+
+		if (supplier_sync_state_enabled)
+			__device_links_supplier_sync_state(link->supplier);
 	}
 
 	dev->links.status = DL_DEV_DRIVER_BOUND;
drivers/of/platform.c +9 -0
index 8d690fa0f47c..24851dcdab55 100644
@@ -617,6 +617,15 @@ static int __init of_platform_default_populate_init(void)
 	return 0;
 }
 arch_initcall_sync(of_platform_default_populate_init);
+
+static int __init of_platform_sync_state_init(void)
+{
+	device_links_supplier_sync_state_enable();
+	bus_for_each_dev(&platform_bus_type, NULL, NULL,
+			 device_links_supplier_sync_state);
+	return 0;
+}
+late_initcall_sync(of_platform_sync_state_init);
 #endif
 
 int of_platform_device_destroy(struct device *dev, void *data)
include/linux/device.h +19 -0
index 026dd842e511..60255f10dffa 100644
@@ -79,6 +79,13 @@ extern void bus_remove_file(struct bus_type *, struct bus_attribute *);
  *		that generate uevents to add the environment variables.
  * @probe:	Called when a new device or driver add to this bus, and callback
  *		the specific driver's probe to initial the matched device.
+ * @sync_state:	Called to sync device state to software state after all the
+ *		state tracking consumers linked to this device (present at
+ *		the time of late_initcall) have successfully bound to a
+ *		driver. If the device has no consumers, this function will
+ *		be called at late_initcall_sync level. If the device has
+ *		consumers that are never bound to a driver, this function
+ *		will never get called until they do.
  * @remove:	Called when a device removed from this bus.
  * @shutdown:	Called at shut-down time to quiesce the device.
  *
@@ -122,6 +129,7 @@ struct bus_type {
 	int (*match)(struct device *dev, struct device_driver *drv);
 	int (*uevent)(struct device *dev, struct kobj_uevent_env *env);
 	int (*probe)(struct device *dev);
+	void (*sync_state)(struct device *dev);
 	int (*remove)(struct device *dev);
 	void (*shutdown)(struct device *dev);
 
@@ -251,6 +259,13 @@ enum probe_type {
  * @probe:	Called to query the existence of a specific device,
  *		whether this driver can work with it, and bind the driver
  *		to a specific device.
+ * @sync_state:	Called to sync device state to software state after all the
+ *		state tracking consumers linked to this device (present at
+ *		the time of late_initcall) have successfully bound to a
+ *		driver. If the device has no consumers, this function will
+ *		be called at late_initcall_sync level. If the device has
+ *		consumers that are never bound to a driver, this function
+ *		will never get called until they do.
  * @remove:	Called when the device is removed from the system to
  *		unbind a device from this driver.
  * @shutdown:	Called at shut-down time to quiesce the device.
@@ -288,6 +303,7 @@ struct device_driver {
 	const struct acpi_device_id	*acpi_match_table;
 
 	int (*probe) (struct device *dev);
+	void (*sync_state)(struct device *dev);
 	int (*remove) (struct device *dev);
 	void (*shutdown) (struct device *dev);
 	int (*suspend) (struct device *dev, pm_message_t state);
@@ -1059,6 +1075,7 @@ struct device {
 	bool			offline_disabled:1;
 	bool			offline:1;
 	bool			of_node_reused:1;
+	bool			state_synced:1;
 #if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_DEVICE) || \
     defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU) || \
     defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU_ALL)
@@ -1401,6 +1418,8 @@ void device_link_remove(void *consumer, struct device *supplier);
 void device_link_wait_for_supplier(struct device *consumer);
 void device_link_check_waiting_consumers(
 		int (*add_suppliers)(struct device *consumer));
+int device_links_supplier_sync_state(struct device *dev, void *data);
+void device_links_supplier_sync_state_enable(void);
 
 #ifndef dev_fmt
 #define dev_fmt(fmt) fmt
Signature · 1 line
2.22.0.410.gd8fdbe21b5-goog