Skip to contentSkip to conversation

Solve postboot supplier cleanup and optimize probe ordering

PATCH In review Saravana Kannan 3 participants 15 messages Sep 4, 2019 – Oct 16, 2019

Subsystems (5)
lore
Share & tools
lore
#0 lore raw

v11 cover Solve postboot supplier cleanup and optimize probe ordering

Add device-links to track functional dependencies between devices after they are created (but before they are probed) by looking at their common DT bindings like clocks, interconnects, etc.

Having functional dependencies automatically added before the devices are probed, provides the following benefits:

- Optimizes device probe order and avoids the useless work of attempting probes of devices that will not probe successfully (because their suppliers aren't present or haven't probed yet).

For example, in a commonly available mobile SoC, registering just one consumer device's driver at an initcall level earlier than the supplier device's driver causes 11 failed probe attempts before the consumer device probes successfully. This was with a kernel with all the drivers statically compiled in. This problem gets a lot worse if all the drivers are loaded as modules without direct symbol dependencies.

- Supplier devices like clock providers, interconnect providers, etc need to keep the resources they provide active and at a particular state(s) during boot up even if their current set of consumers don't request the resource to be active. This is because the rest of the consumers might not have probed yet and turning off the resource before all the consumers have probed could lead to a hang or undesired user experience.

Some frameworks (Eg: regulator) handle this today by turning off "unused" resources at late_initcall_sync and hoping all the devices have probed by then. This is not a valid assumption for systems with loadable modules. Other frameworks (Eg: clock) just don't handle this due to the lack of a clear signal for when they can turn off resources. This leads to downstream hacks to handle cases like this that can easily be solved in the upstream kernel.

By linking devices before they are probed, we give suppliers a clear count of the number of dependent consumers. Once all of the consumers are active, the suppliers can turn off the unused resources without making assumptions about the number of consumers.

By default we just add device-links to track "driver presence" (probe succeeded) of the supplier device. If any other functionality provided by device-links are needed, it is left to the consumer/supplier devices to change the link when they probe.

v1 -> v2:
- Drop patch to speed up of_find_device_by_node()
- Drop depends-on property and use existing bindings

v2 -> v3:
- Refactor the code to have driver core initiate the linking of devs
- Have driver core link consumers to supplier before it's probed
- Add support for drivers to edit the device links before probing

v3 -> v4:
- Tested edit_links() on system with cyclic dependency. Works.
- Added some checks to make sure device link isn't attempted from parent device node to child device node.
- Added way to pause/resume sync_state callbacks across of_platform_populate().
- Recursively parse DT node to create device links from parent to suppliers of parent and all child nodes.

v4 -> v5:
- Fixed copy-pasta bugs with linked list handling
- Walk up the phandle reference till I find an actual device (needed for regulators to work)
- Added support for linking devices from regulator DT bindings
- Tested the whole series again to make sure cyclic dependencies are broken with edit_links() and regulator links are created properly.

v5 -> v6:
- Split, squashed and reordered some of the patches.
- Refactored the device linking code to follow the same code pattern for any property.

v6 -> v7:
- No functional changes.
- Renamed i to index
- Added comment to clarify not having to check property name for every index
- Added "matched" variable to clarify code. No functional change.
- Added comments to include/linux/device.h for add_links()

v7 -> v8:
- Rebased on top of linux-next to handle device link changes in [1]

v8 -> v9:
- Fixed kbuild test bot reported errors (docs and const)

v9->v10:
- Changes made based on reviews on LKML [2] and discussions at ELC [3]
- Dropped the edit_links() patch
- Dropped the patch that skips linking for default bus nodes
- 1/7: Changed from bus.add_links() to fwnode.ops.add_links()
- 1/7: Update device link doc
- 1/7: Lots of comments/fn doc updates
- 1/7: Renamed device_link_check_waiting_consumers() to device_link_add_missing_supplier_links()
- 2/7: Moved DT parsing/linking code from of/platform.c to of/property.c
- 2/7: Lots of comments/fn doc updates
- 2/7: Returned errors for all error cases in of_link_to_phandle()
- 2/7: Some minor code refactor to remove "bool done"
- 2/7: Added debug messages when links not created due permanent errors
- 3/7: Minor comments update
- Added 2 new patches 6/7 and 7/7 to handle cyclic dependencies using depends-on

v10->v11:
- Dropped 6/7 and 7/7 from previous series that tried to handle cycles in DT dependencies. We can solve it later when we actually hit a real world issue in DT.
- Added a new 1/7 that shifts the numbering for the rest of the patches
- 1/7 adds a way to look up a device from a fwnode so that this series can work across bus and firmware types
- 3/7 removed references to platform_device from of/property.c
- 4/7 Minor variable rename
- 4/7 Defer sync_state() be default at driver core level and resume at late_initcall_sync(). That way, we don't depend on any specific bus types having to pause/resume sync_state() till late_initcall_sync()

[1] - https://lore.kernel.org/lkml/2305283.AStDPdUUnE@kreacher/
[2] - https://lore.kernel.org/lkml/[email protected]/
[3] - https://lore.kernel.org/lkml/CAGETcx_pSnC_2D7ufLRyfE3b8uRc814XEf8zu+SpNtT7_Z8NLg@mail.gmail.com/

-Saravana

Saravana Kannan (6):
driver core: Add fwnode_to_dev() to look up device from fwnode driver core: Add support for linking devices during device addition
of: property: Add functional dependency link from DT bindings driver core: Add sync_state driver/bus callback of/platform: Pause/resume sync state during init and
of_platform_populate()
of: property: Create device links for all child-supplier depencencies

 .../admin-guide/kernel-parameters.rst         |   1 +
 .../admin-guide/kernel-parameters.txt         |   6 +
 Documentation/driver-api/device_link.rst      |   3 +-
 drivers/base/core.c                           | 167 ++++++++++++
 drivers/of/platform.c                         |  12 +
 drivers/of/property.c                         | 245 ++++++++++++++++++
 include/linux/device.h                        |  26 ++
 include/linux/fwnode.h                        |  19 ++
 8 files changed, 478 insertions(+), 1 deletion(-)
Signature · 1 line
2.23.0.187.g17f5b7556c-goog
#1 lore raw

PATCH v11 1/6 driver core: Add fwnode_to_dev() to look up device from fwnode

It's often useful to look up a device that corresponds to a fwnode. So add an API to do that irrespective of the bus on which the device has been added to.

---
 drivers/base/core.c    | 7 +++++++
 include/linux/fwnode.h | 2 ++
 2 files changed, 9 insertions(+)
drivers/base/core.c +7 -0
index 2db62d98e395..510fabf8918c 100644
@@ -2198,6 +2198,10 @@ int device_add(struct device *dev)
 					     BUS_NOTIFY_ADD_DEVICE, dev);
 
 	kobject_uevent(&dev->kobj, KOBJ_ADD);
+
+	if (dev->fwnode && !dev->fwnode->dev)
+		dev->fwnode->dev = dev;
+
 	bus_probe_device(dev);
 	if (parent)
 		klist_add_tail(&dev->p->knode_parent,
@@ -2342,6 +2346,9 @@ void device_del(struct device *dev)
 	kill_device(dev);
 	device_unlock(dev);
 
+	if (dev->fwnode && dev->fwnode->dev == dev)
+		dev->fwnode->dev = NULL;
+
 	/* Notify clients of device removal.  This call must come
 	 * before dpm_sysfs_remove().
 	 */
include/linux/fwnode.h +2 -0
index ababd6bc82f3..d8c6d231d577 100644
@@ -17,6 +17,7 @@ struct device;
 struct fwnode_handle {
 	struct fwnode_handle *secondary;
 	const struct fwnode_operations *ops;
+	struct device *dev;
 };
 
 /**
@@ -123,5 +124,6 @@ struct fwnode_operations {
 		if (fwnode_has_op(fwnode, op))				\
 			(fwnode)->ops->op(fwnode, ## __VA_ARGS__);	\
 	} while (false)
+#define get_dev_from_fwnode(fwnode)	get_device((fwnode)->dev)
 
 #endif
Signature · 1 line
2.23.0.187.g17f5b7556c-goog
#2 lore raw

PATCH v11 2/6 driver core: Add support for linking devices during device addition

In reply to Saravana Kannan

The firmware corresponding to a device (dev.fwnode) might be able to provide functional dependency information between a device and its supplier and consumer devices. Tracking this functional dependency allows optimizing device probe order and informing a supplier when all its consumers have probed (and thereby actively managing their resources).

The existing device links feature allows tracking and using supplier-consumer relationships. So, this patch adds the add_links() fwnode callback to allow firmware to create device links for each device as the device is added.

However, when consumer devices are added, they might not have a supplier device to link to despite needing mandatory resources/functionality from one or more suppliers. A waiting_for_suppliers list is created to track such consumers and retry linking them when new devices get added.

---
 Documentation/driver-api/device_link.rst |  3 +-
 drivers/base/core.c                      | 88 ++++++++++++++++++++++++
 include/linux/device.h                   |  2 +
 include/linux/fwnode.h                   | 17 +++++
 4 files changed, 109 insertions(+), 1 deletion(-)
Documentation/driver-api/device_link.rst +2 -1
index 1b5020ec6517..bc2d89af88ce 100644
@@ -281,7 +281,8 @@ State machine
   :c:func:`driver_bound()`.)
 
 * Before a consumer device is probed, presence of supplier drivers is
-  verified by checking that links to suppliers are in ``DL_STATE_AVAILABLE``
+  verified by checking the consumer device is not in the wait_for_suppliers
+  list and by checking that links to suppliers are in ``DL_STATE_AVAILABLE``
   state.  The state of the links is updated to ``DL_STATE_CONSUMER_PROBE``.
   (Call to :c:func:`device_links_check_suppliers()` from
   :c:func:`really_probe()`.)
drivers/base/core.c +88 -0

124 lines not on this page · raw message

include/linux/device.h +2 -0
index f30e80185825..968316bb3bd1 100644
@@ -1136,11 +1136,13 @@ enum dl_dev_state {
  * struct dev_links_info - Device data related to device links.
  * @suppliers: List of links to supplier devices.
  * @consumers: List of links to consumer devices.
+ * @needs_suppliers: Hook to global list of devices waiting for suppliers.
  * @status: Driver status information.
  */
 struct dev_links_info {
 	struct list_head suppliers;
 	struct list_head consumers;
+	struct list_head needs_suppliers;
 	enum dl_dev_state status;
 };
 
include/linux/fwnode.h +17 -0
index d8c6d231d577..6ae05b9ce359 100644
@@ -66,6 +66,21 @@ struct fwnode_reference_args {
  *			       endpoint node.
  * @graph_get_port_parent: Return the parent node of a port node.
  * @graph_parse_endpoint: Parse endpoint for port and endpoint id.
+ * @add_links:	Called after the device corresponding to the fwnode is added
+ *		using device_add(). The function is expected to create device
+ *		links to all the suppliers of the device that are available at
+ *		the time this function is called.  The function must NOT stop
+ *		at the first failed device link if other unlinked supplier
+ *		devices are present in the system.  If some suppliers are not
+ *		yet available, this function will be called again when other
+ *		devices are added to allow creating device links to any newly
+ *		available suppliers.
+ *
+ *		Return 0 if device links have been successfully created to all
+ *		the suppliers of this device or if the supplier information is
+ *		not known. Return an error if and only if the supplier
+ *		information is known but some of the suppliers are not yet
+ *		available to create device links to.
  */
 struct fwnode_operations {
 	struct fwnode_handle *(*get)(struct fwnode_handle *fwnode);
@@ -103,6 +118,8 @@ struct fwnode_operations {
 	(*graph_get_port_parent)(struct fwnode_handle *fwnode);
 	int (*graph_parse_endpoint)(const struct fwnode_handle *fwnode,
 				    struct fwnode_endpoint *endpoint);
+	int (*add_links)(const struct fwnode_handle *fwnode,
+			 struct device *dev);
 };
 
 #define fwnode_has_op(fwnode, op)				\
Signature · 1 line
2.23.0.187.g17f5b7556c-goog
#3 lore raw

PATCH v11 3/6 of: property: Add functional dependency link from DT bindings

In reply to Saravana Kannan

Add device links after the devices are created (but before they are probed) by looking at common DT bindings like clocks and interconnects.

Automatically adding device links for functional dependencies at the framework level provides the following benefits:

- Optimizes device probe order and avoids the useless work of attempting probes of devices that will not probe successfully (because their suppliers aren't present or haven't probed yet).

For example, in a commonly available mobile SoC, registering just one consumer device's driver at an initcall level earlier than the supplier device's driver causes 11 failed probe attempts before the consumer device probes successfully. This was with a kernel with all the drivers statically compiled in. This problem gets a lot worse if all the drivers are loaded as modules without direct symbol dependencies.

- Supplier devices like clock providers, interconnect providers, etc need to keep the resources they provide active and at a particular state(s) during boot up even if their current set of consumers don't request the resource to be active. This is because the rest of the consumers might not have probed yet and turning off the resource before all the consumers have probed could lead to a hang or undesired user experience.

Some frameworks (Eg: regulator) handle this today by turning off "unused" resources at late_initcall_sync and hoping all the devices have probed by then. This is not a valid assumption for systems with loadable modules. Other frameworks (Eg: clock) just don't handle this due to the lack of a clear signal for when they can turn off resources. This leads to downstream hacks to handle cases like this that can easily be solved in the upstream kernel.

By linking devices before they are probed, we give suppliers a clear count of the number of dependent consumers. Once all of the consumers are active, the suppliers can turn off the unused resources without making assumptions about the number of consumers.

By default we just add device-links to track "driver presence" (probe succeeded) of the supplier device. If any other functionality provided by device-links are needed, it is left to the consumer/supplier devices to change the link when they probe.

kbuild test robot reported clang error about missing const

---
 .../admin-guide/kernel-parameters.rst         |   1 +
 .../admin-guide/kernel-parameters.txt         |   6 +
 drivers/of/property.c                         | 241 ++++++++++++++++++
 3 files changed, 248 insertions(+)
Documentation/admin-guide/kernel-parameters.rst +1 -0
index d05d531b4ec9..6d421694d98e 100644
@@ -127,6 +127,7 @@ parameter is applicable::
 	NET	Appropriate network support is enabled.
 	NUMA	NUMA support is enabled.
 	NFS	Appropriate NFS support is enabled.
+	OF	Devicetree is enabled.
 	OSS	OSS sound support is enabled.
 	PV_OPS	A paravirtualized kernel is enabled.
 	PARIDE	The ParIDE (parallel port IDE) subsystem is enabled.
Documentation/admin-guide/kernel-parameters.txt +6 -0
index af0a62af6fd8..e95f2a58acc5 100644
@@ -3181,6 +3181,12 @@
 			This can be set from sysctl after boot.
 			See Documentation/admin-guide/sysctl/vm.rst for details.
 
+	of_devlink	[OF, KNL] Create device links between consumer and
+			supplier devices by scanning the devictree to infer the
+			consumer/supplier relationships.  A consumer device
+			will not be probed until all the supplier devices have
+			probed successfully.
+
 	ohci1394_dma=early	[HW] enable debugging via the ohci1394 driver.
 			See Documentation/debugging-via-ohci1394.txt for more
 			info.
drivers/of/property.c +241 -0

258 lines not on this page · raw message

Signature · 1 line
2.23.0.187.g17f5b7556c-goog
#4 lore raw

Re: [PATCH v11 3/6] of: property: Add functional dependency link from DT bindings

Quoting Saravana Kannan (2019-09-04 14:11:22)

Quoted 27 lines

Add device links after the devices are created (but before they are probed) by looking at common DT bindings like clocks and interconnects.

Automatically adding device links for functional dependencies at the framework level provides the following benefits:

- Optimizes device probe order and avoids the useless work of attempting probes of devices that will not probe successfully (because their suppliers aren't present or haven't probed yet).

For example, in a commonly available mobile SoC, registering just one consumer device's driver at an initcall level earlier than the supplier device's driver causes 11 failed probe attempts before the consumer device probes successfully. This was with a kernel with all the drivers statically compiled in. This problem gets a lot worse if all the drivers are loaded as modules without direct symbol dependencies.

- Supplier devices like clock providers, interconnect providers, etc need to keep the resources they provide active and at a particular state(s) during boot up even if their current set of consumers don't request the resource to be active. This is because the rest of the consumers might not have probed yet and turning off the resource before all the consumers have probed could lead to a hang or undesired user experience.

Quoted 6 lines

Some frameworks (Eg: regulator) handle this today by turning off "unused" resources at late_initcall_sync and hoping all the devices have probed by then. This is not a valid assumption for systems with loadable modules. Other frameworks (Eg: clock) just don't handle this due to the lack of a clear signal for when they can turn off resources.

The clk framework disables unused clks at late_initcall_sync. What do you mean clk framework doesn't turn them off because of a clear signal?

Quoted 8 lines

This leads to downstream hacks to handle cases like this
that can easily be solved in the upstream kernel.

By linking devices before they are probed, we give suppliers a clear count of the number of dependent consumers. Once all of the consumers are active, the suppliers can turn off the unused resources without making assumptions about the number of consumers.

Quoted 5 lines

By default we just add device-links to track "driver presence" (probe succeeded) of the supplier device. If any other functionality provided by device-links are needed, it is left to the consumer/supplier devices to change the link when they probe.

Are there tabs above? Indentation looks off.

Documentation/admin-guide/kernel-parameters.rst parameter is applicable:: · line 133

Quoted 12 lines
Documentation/admin-guide/kernel-parameters.rst +1 -0
index d05d531b4ec9..6d421694d98e 100644
@@ -127,6 +127,7 @@ parameter is applicable::
        NET     Appropriate network support is enabled.
        NUMA    NUMA support is enabled.
        NFS     Appropriate NFS support is enabled.
+       OF      Devicetree is enabled.
        OSS     OSS sound support is enabled.
        PV_OPS  A paravirtualized kernel is enabled.
        PARIDE  The ParIDE (parallel port IDE) subsystem is enabled.

This could be split off and applied for dt_cpu_ftrs= in Documentation/admin-guide/kernel-parameters.txt

Quoted 13 lines
drivers/of/property.c +1 -0
index d7fa75e31f22..23b5ee5b0570 100644
@@ -25,6 +25,7 @@
 #include <linux/of_device.h>
 #include <linux/of_graph.h>
 #include <linux/string.h>
+#include <linux/moduleparam.h>
 
 #include "of_private.h"
 

drivers/of/property.c of_fwnode_device_get_match_data(const struct fwnode_handle *fwnode, · line 989

Quoted 4 lines
drivers/of/property.c +1 -0
@@ -985,6 +986,245 @@ of_fwnode_device_get_match_data(const struct fwnode_handle *fwnode,
        return of_device_get_match_data(dev);
 }
 
+static bool of_is_ancestor_of(struct device_node *test_ancestor,

Maybe add kernel-doc indicating that this function returns with the of_node requiring an of_node_put() call on it, unless it returns false in which case it doesn't?

Quoted 14 lines
drivers/of/property.c +14 -0
+                             struct device_node *child)
+{
+       of_node_get(child);
+       while (child) {
+               if (child == test_ancestor) {
+                       of_node_put(child);
+                       return false;
+               }
+               child = of_get_next_parent(child);
+       }
+       return true;
+}
+
+/**

drivers/of/property.c of_fwnode_device_get_match_data(const struct fwnode_handle *fwnode, · line 1013

Quoted 10 lines
drivers/of/property.c +10 -0
+ * of_link_to_phandle - Add device link to supplier from supplier phandle
+ * @dev: consumer device
+ * @sup_np: phandle to supplier device tree node
+ *
+ * Given a phandle to a supplier device tree node (@sup_np), this function
+ * finds the device that owns the supplier device tree node and creates a
+ * device link from @dev consumer device to the supplier device. This function
+ * doesn't create device links for invalid scenarios such as trying to create a
+ * link with a parent device as the consumer of its child device. In such
+ * cases, it returns an error.

Doesn't device links have problems with making cycles between providers and consumers? We have some scenarios where two clk providers are consumers of each other but it isn't a parent child relationship. They're peers on the SoC bus and there isn't a cycle in the clk tree but there is a cycle between the two device nodes and providers. I don't see the avoidance here but maybe I missed something?

drivers/of/property.c of_fwnode_device_get_match_data(const struct fwnode_handle *fwnode, · line 1024

Quoted 11 lines
drivers/of/property.c +11 -0
+ *
+ * Returns:
+ * - 0 if link successfully created to supplier
+ * - -EAGAIN if linking to the supplier should be reattempted
+ * - -EINVAL if the supplier link is invalid and should not be created
+ * - -ENODEV if there is no device that corresponds to the supplier phandle
+ */
+static int of_link_to_phandle(struct device *dev, struct device_node *sup_np)
+{
+       struct device *sup_dev;
+       u32 dl_flags = DL_FLAG_AUTOPROBE_CONSUMER;

Is it really a u32 instead of an unsigned int or unsigned long?

drivers/of/property.c of_fwnode_device_get_match_data(const struct fwnode_handle *fwnode, · line 1034

Quoted 10 lines
drivers/of/property.c +10 -0
+       int ret = 0;
+       struct device_node *tmp_np = sup_np;
+
+       of_node_get(sup_np);
+       /*
+        * Find the device node that contains the supplier phandle.  It may be
+        * @sup_np or it may be an ancestor of @sup_np.
+        */
+       while (sup_np && !of_find_property(sup_np, "compatible", NULL))
+               sup_np = of_get_next_parent(sup_np);

I don't get this. This is assuming that drivers are only probed for device nodes that have a compatible string? What about drivers that make sub-devices for clk support that have drivers in drivers/clk/ that then attach at runtime later? This happens sometimes for MFDs that want to split the functionality across the driver tree to the respective subsystems.

Quoted 56 lines
drivers/of/property.c +56 -0
+       if (!sup_np) {
+               dev_dbg(dev, "Not linking to %pOFP - No device\n", tmp_np);
+               return -ENODEV;
+       }
+
+       /*
+        * Don't allow linking a device node as a consumer of one of its
+        * descendant nodes. By definition, a child node can't be a functional
+        * dependency for the parent node.
+        */
+       if (!of_is_ancestor_of(dev->of_node, sup_np)) {
+               dev_dbg(dev, "Not linking to %pOFP - is descendant\n", sup_np);
+               of_node_put(sup_np);
+               return -EINVAL;
+       }
+       sup_dev = get_dev_from_fwnode(&sup_np->fwnode);
+       of_node_put(sup_np);
+       if (!sup_dev)
+               return -EAGAIN;
+       if (!device_link_add(dev, sup_dev, dl_flags))
+               ret = -EAGAIN;
+       put_device(sup_dev);
+       return ret;
+}
+
+/**
+ * parse_prop_cells - Property parsing function for suppliers
+ *
+ * @np:                Pointer to device tree node containing a list
+ * @prop_name: Name of property to be parsed. Expected to hold phandle values
+ * @index:     For properties holding a list of phandles, this is the index
+ *             into the list.
+ * @list_name: Property name that is known to contain list of phandle(s) to
+ *             supplier(s)
+ * @cells_name:        property name that specifies phandles' arguments count
+ *
+ * This is a helper function to parse properties that have a known fixed name
+ * and are a list of phandles and phandle arguments.
+ *
+ * Returns:
+ * - phandle node pointer with refcount incremented. Caller must of_node_put()
+ *   on it when done.
+ * - NULL if no phandle found at index
+ */
+static struct device_node *parse_prop_cells(struct device_node *np,
+                                           const char *prop_name, int index,
+                                           const char *list_name,
+                                           const char *cells_name)
+{
+       struct of_phandle_args sup_args;
+
+       if (strcmp(prop_name, list_name))
+               return NULL;
+
+       if (of_parse_phandle_with_args(np, list_name, cells_name, index,
+                                      &sup_args))

drivers/of/property.c of_fwnode_device_get_match_data(const struct fwnode_handle *fwnode, · line 1100

Quoted 10 lines
drivers/of/property.c +10 -0
+               return NULL;
+
+       return sup_args.np;
+}
+
+static struct device_node *parse_clocks(struct device_node *np,
+                                       const char *prop_name, int index)
+{
+       return parse_prop_cells(np, prop_name, index, "clocks", "#clock-cells");
+}

Can this use of_parse_clkspec() instead? If it is exported out of the clk framework (which is weird to me for other reasons) then it should work to call that with the index passed in to this function. Ideally we don't have more than one place where we parse clock specifiers for a node.

Another question is what happens for devices that are in DT but are using "clock-ranges"? As far as I know there are some DTS files that use that property to only send the clocks to some bus node that then lets devices find the "clocks" and "clock-names" properties from the bus node instead of from the node that corresponds to their device.

drivers/of/property.c of_fwnode_device_get_match_data(const struct fwnode_handle *fwnode, · line 1109

Quoted 9 lines
drivers/of/property.c +9 -0
+
+static struct device_node *parse_interconnects(struct device_node *np,
+                                              const char *prop_name, int index)
+{
+       return parse_prop_cells(np, prop_name, index, "interconnects",
+                               "#interconnect-cells");
+}
+
+static int strcmp_suffix(const char *str, const char *suffix)

And this doesn't go to lib/string.c why?

Quoted 18 lines
drivers/of/property.c +18 -0
+{
+       unsigned int len, suffix_len;
+
+       len = strlen(str);
+       suffix_len = strlen(suffix);
+       if (len <= suffix_len)
+               return -1;
+       return strcmp(str + len - suffix_len, suffix);
+}
+
+static struct device_node *parse_regulators(struct device_node *np,
+                                           const char *prop_name, int index)
+{
+       if (index || strcmp_suffix(prop_name, "-supply"))
+               return NULL;
+
+       return of_parse_phandle(np, prop_name, 0);
+}

drivers/of/property.c of_fwnode_device_get_match_data(const struct fwnode_handle *fwnode, · line 1137

Quoted 10 lines
drivers/of/property.c +10 -0
+
+/**
+ * struct supplier_bindings - Property parsing functions for suppliers
+ *
+ * @parse_prop: function name
+ *     parse_prop() finds the node corresponding to a supplier phandle
+ * @parse_prop.np: Pointer to device node holding supplier phandle property
+ * @parse_prop.prop_name: Name of property holding a phandle value
+ * @parse_prop.index: For properties holding a list of phandles, this is the
+ *                   index into the list

This is interesting kernel-doc. I've never seen it before. Does it work?

drivers/of/property.c of_fwnode_device_get_match_data(const struct fwnode_handle *fwnode, · line 1147

Quoted 10 lines
drivers/of/property.c +10 -0
+ *
+ * Returns:
+ * parse_prop() return values are
+ * - phandle node pointer with refcount incremented. Caller must of_node_put()
+ *   on it when done.
+ * - NULL if no phandle found at index
+ */
+struct supplier_bindings {
+       struct device_node *(*parse_prop)(struct device_node *np,
+                                         const char *prop_name, int index);

Maybe this should be a typedef instead of a struct unless you plan to put more members in this struct? Or are arrays of function pointers impossible?

drivers/of/property.c of_fwnode_device_get_match_data(const struct fwnode_handle *fwnode, · line 1150

Quoted 3 lines
drivers/of/property.c +3 -0
+};
+
+static const struct supplier_bindings bindings[] = {

This variable name is really bad. Please make it much more specific to this file instead of being called 'bindings' so that grepping for it and looking for it in kallsyms isn't difficult.

drivers/of/property.c of_fwnode_device_get_match_data(const struct fwnode_handle *fwnode, · line 1154

Quoted 4 lines
drivers/of/property.c +4 -0
+       { .parse_prop = parse_clocks, },
+       { .parse_prop = parse_interconnects, },
+       { .parse_prop = parse_regulators, },
+       {},

Nitpick: Don't put a comma after the sentinel so that it causes a compile error to follow it with another "valid" entry.

Quoted 27 lines
drivers/of/property.c +27 -0
+};
+
+/**
+ * of_link_property - Create device links to suppliers listed in a property
+ * @dev: Consumer device
+ * @con_np: The consumer device tree node which contains the property
+ * @prop_name: Name of property to be parsed
+ *
+ * This function checks if the property @prop_name that is present in the
+ * @con_np device tree node is one of the known common device tree bindings
+ * that list phandles to suppliers. If @prop_name isn't one, this function
+ * doesn't do anything.
+ *
+ * If @prop_name is one, this function attempts to create device links from the
+ * consumer device @dev to all the devices of the suppliers listed in
+ * @prop_name.
+ *
+ * Any failed attempt to create a device link will NOT result in an immediate
+ * return.  of_link_property() must create links to all the available supplier
+ * devices even when attempts to create a link to one or more suppliers fail.
+ */
+static int of_link_property(struct device *dev, struct device_node *con_np,
+                            const char *prop_name)
+{
+       struct device_node *phandle;
+       const struct supplier_bindings *s = bindings;
+       unsigned int i = 0;

drivers/of/property.c of_fwnode_device_get_match_data(const struct fwnode_handle *fwnode, · line 1191

Quoted 10 lines
drivers/of/property.c +10 -0
+       bool matched = false;
+       int ret = 0;
+
+       /* Do not stop at first failed link, link all available suppliers. */
+       while (!matched && s->parse_prop) {
+               while ((phandle = s->parse_prop(con_np, prop_name, i))) {
+                       matched = true;
+                       i++;
+                       if (of_link_to_phandle(dev, phandle) == -EAGAIN)
+                               ret = -EAGAIN;

And don't break?

drivers/of/property.c of_fwnode_device_get_match_data(const struct fwnode_handle *fwnode, · line 1199

Quoted 8 lines
drivers/of/property.c +8 -0
+                       of_node_put(phandle);
+               }
+               s++;
+       }
+       return ret;
+}
+
+static int __of_link_to_suppliers(struct device *dev,

Why the double underscore?

drivers/of/property.c of_fwnode_device_get_match_data(const struct fwnode_handle *fwnode, · line 1208

Quoted 9 lines
drivers/of/property.c +9 -0
+                                 struct device_node *con_np)
+{
+       struct device_node *child;
+       struct property *p;
+       int ret = 0;
+
+       for_each_property_of_node(con_np, p)
+               if (of_link_property(dev, con_np, p->name))
+                       ret = -EAGAIN;

Same comment.

Quoted 3 lines
drivers/of/property.c +3 -0
+
+       return ret;
+}
#5 lore raw

Re: [PATCH v11 3/6] of: property: Add functional dependency link from DT bindings

Quoted 35 lines · Stephen Boyd

Quoting Saravana Kannan (2019-09-04 14:11:22)

Quoted 27 lines

Add device links after the devices are created (but before they are probed) by looking at common DT bindings like clocks and interconnects.

Automatically adding device links for functional dependencies at the framework level provides the following benefits:

- Optimizes device probe order and avoids the useless work of attempting probes of devices that will not probe successfully (because their suppliers aren't present or haven't probed yet).

For example, in a commonly available mobile SoC, registering just one consumer device's driver at an initcall level earlier than the supplier device's driver causes 11 failed probe attempts before the consumer device probes successfully. This was with a kernel with all the drivers statically compiled in. This problem gets a lot worse if all the drivers are loaded as modules without direct symbol dependencies.

- Supplier devices like clock providers, interconnect providers, etc need to keep the resources they provide active and at a particular state(s) during boot up even if their current set of consumers don't request the resource to be active. This is because the rest of the consumers might not have probed yet and turning off the resource before all the consumers have probed could lead to a hang or undesired user experience.

Quoted 6 lines

Some frameworks (Eg: regulator) handle this today by turning off "unused" resources at late_initcall_sync and hoping all the devices have probed by then. This is not a valid assumption for systems with loadable modules. Other frameworks (Eg: clock) just don't handle this due to the lack of a clear signal for when they can turn off resources.

Quoted 2 lines

The clk framework disables unused clks at late_initcall_sync. What do you mean clk framework doesn't turn them off because of a clear signal?

There's a number of minor things you pointed out in this review.

Saravana, can you send a follow-on patch for the minor code cleanups like formatting and the like that was found here?

Quoted 18 lines

drivers/of/property.c of_fwnode_device_get_match_data(const struct fwnode_handle *fwnode, · line 1024

Quoted 4 lines
drivers/of/property.c +4 -0
+static int of_link_to_phandle(struct device *dev, struct device_node *sup_np)
+{
+       struct device *sup_dev;
+       u32 dl_flags = DL_FLAG_AUTOPROBE_CONSUMER;

Is it really a u32 instead of an unsigned int or unsigned long?

drivers/of/property.c of_fwnode_device_get_match_data(const struct fwnode_handle *fwnode, · line 1034

Quoted 10 lines
drivers/of/property.c +10 -0
+       int ret = 0;
+       struct device_node *tmp_np = sup_np;
+
+       of_node_get(sup_np);
+       /*
+        * Find the device node that contains the supplier phandle.  It may be
+        * @sup_np or it may be an ancestor of @sup_np.
+        */
+       while (sup_np && !of_find_property(sup_np, "compatible", NULL))
+               sup_np = of_get_next_parent(sup_np);
Quoted 6 lines

I don't get this. This is assuming that drivers are only probed for device nodes that have a compatible string? What about drivers that make sub-devices for clk support that have drivers in drivers/clk/ that then attach at runtime later? This happens sometimes for MFDs that want to split the functionality across the driver tree to the respective subsystems.

For that, the link would not be there, correct?

Quoted 10 lines
Quoted 10 lines
drivers/of/property.c +10 -0
+static int of_link_property(struct device *dev, struct device_node *con_np,
+                            const char *prop_name)
+{
+       struct device_node *phandle;
+       const struct supplier_bindings *s = bindings;
+       unsigned int i = 0;
+       bool matched = false;
+       int ret = 0;
+
+       /* Do not stop at first failed link, link all available suppliers. */
Quoted 8 lines

drivers/of/property.c of_fwnode_device_get_match_data(const struct fwnode_handle *fwnode, · line 1191

Quoted 6 lines
drivers/of/property.c +6 -0
+       while (!matched && s->parse_prop) {
+               while ((phandle = s->parse_prop(con_np, prop_name, i))) {
+                       matched = true;
+                       i++;
+                       if (of_link_to_phandle(dev, phandle) == -EAGAIN)
+                               ret = -EAGAIN;

And don't break?

There was comments before about how this is not needed. Frank asked that the comment be removed. And now you point it out again :)

Look at the comment a few lines up, we have to go through all of the suppliers.

Quoted 4 lines
Quoted 1 line

+static int __of_link_to_suppliers(struct device *dev,

Why the double underscore?

Quoted 11 lines

drivers/of/property.c of_fwnode_device_get_match_data(const struct fwnode_handle *fwnode, · line 1208

Quoted 9 lines
drivers/of/property.c +9 -0
+                                 struct device_node *con_np)
+{
+       struct device_node *child;
+       struct property *p;
+       int ret = 0;
+
+       for_each_property_of_node(con_np, p)
+               if (of_link_property(dev, con_np, p->name))
+                       ret = -EAGAIN;

Same comment.

Same response as above :)

thanks,

greg k-h

#6 lore raw

Re: [PATCH v11 3/6] of: property: Add functional dependency link from DT bindings

Quoted 37 lines · Greg Kroah-Hartman
Quoted 35 lines · Stephen Boyd

Quoting Saravana Kannan (2019-09-04 14:11:22)

Quoted 33 lines

Add device links after the devices are created (but before they are probed) by looking at common DT bindings like clocks and interconnects.

Automatically adding device links for functional dependencies at the framework level provides the following benefits:

- Optimizes device probe order and avoids the useless work of attempting probes of devices that will not probe successfully (because their suppliers aren't present or haven't probed yet).

For example, in a commonly available mobile SoC, registering just one consumer device's driver at an initcall level earlier than the supplier device's driver causes 11 failed probe attempts before the consumer device probes successfully. This was with a kernel with all the drivers statically compiled in. This problem gets a lot worse if all the drivers are loaded as modules without direct symbol dependencies.

- Supplier devices like clock providers, interconnect providers, etc need to keep the resources they provide active and at a particular state(s) during boot up even if their current set of consumers don't request the resource to be active. This is because the rest of the consumers might not have probed yet and turning off the resource before all the consumers have probed could lead to a hang or undesired user experience.

Some frameworks (Eg: regulator) handle this today by turning off "unused" resources at late_initcall_sync and hoping all the devices have probed by then. This is not a valid assumption for systems with loadable modules. Other frameworks (Eg: clock) just don't handle this due to the lack of a clear signal for when they can turn off resources.

Quoted 7 lines
Quoted 2 lines

The clk framework disables unused clks at late_initcall_sync. What do you mean clk framework doesn't turn them off because of a clear signal?

There's a number of minor things you pointed out in this review.

Saravana, can you send a follow-on patch for the minor code cleanups like formatting and the like that was found here?

Will do next week.

Thanks,
Saravana

Quoted 74 lines
Quoted 19 lines

drivers/of/property.c of_fwnode_device_get_match_data(const struct fwnode_handle *fwnode, · line 1024

Quoted 5 lines
drivers/of/property.c +4 -0
+static int of_link_to_phandle(struct device *dev, struct device_node *sup_np)
+{
+       struct device *sup_dev;
+       u32 dl_flags = DL_FLAG_AUTOPROBE_CONSUMER;

Is it really a u32 instead of an unsigned int or unsigned long?

drivers/of/property.c of_fwnode_device_get_match_data(const struct fwnode_handle *fwnode, · line 1034

Quoted 10 lines
drivers/of/property.c +10 -0
+       int ret = 0;
+       struct device_node *tmp_np = sup_np;
+
+       of_node_get(sup_np);
+       /*
+        * Find the device node that contains the supplier phandle.  It may be
+        * @sup_np or it may be an ancestor of @sup_np.
+        */
+       while (sup_np && !of_find_property(sup_np, "compatible", NULL))
+               sup_np = of_get_next_parent(sup_np);
Quoted 6 lines

I don't get this. This is assuming that drivers are only probed for device nodes that have a compatible string? What about drivers that make sub-devices for clk support that have drivers in drivers/clk/ that then attach at runtime later? This happens sometimes for MFDs that want to split the functionality across the driver tree to the respective subsystems.

For that, the link would not be there, correct?

Quoted 10 lines
Quoted 10 lines
drivers/of/property.c +10 -0
+static int of_link_property(struct device *dev, struct device_node *con_np,
+                            const char *prop_name)
+{
+       struct device_node *phandle;
+       const struct supplier_bindings *s = bindings;
+       unsigned int i = 0;
+       bool matched = false;
+       int ret = 0;
+
+       /* Do not stop at first failed link, link all available suppliers. */
Quoted 8 lines

drivers/of/property.c of_fwnode_device_get_match_data(const struct fwnode_handle *fwnode, · line 1191

Quoted 6 lines
drivers/of/property.c +6 -0
+       while (!matched && s->parse_prop) {
+               while ((phandle = s->parse_prop(con_np, prop_name, i))) {
+                       matched = true;
+                       i++;
+                       if (of_link_to_phandle(dev, phandle) == -EAGAIN)
+                               ret = -EAGAIN;

And don't break?

There was comments before about how this is not needed. Frank asked that the comment be removed. And now you point it out again :)

Look at the comment a few lines up, we have to go through all of the suppliers.

Quoted 4 lines
Quoted 1 line

+static int __of_link_to_suppliers(struct device *dev,

Why the double underscore?

Quoted 11 lines

drivers/of/property.c of_fwnode_device_get_match_data(const struct fwnode_handle *fwnode, · line 1208

Quoted 9 lines
drivers/of/property.c +9 -0
+                                 struct device_node *con_np)
+{
+       struct device_node *child;
+       struct property *p;
+       int ret = 0;
+
+       for_each_property_of_node(con_np, p)
+               if (of_link_property(dev, con_np, p->name))
+                       ret = -EAGAIN;

Same comment.

Same response as above :)

thanks,

greg k-h

#7 lore raw

Re: [PATCH v11 3/6] of: property: Add functional dependency link from DT bindings

In reply to Greg Kroah-Hartman

Quoting Greg Kroah-Hartman (2019-10-04 08:37:50)

Quoted 13 lines · Stephen Boyd
Quoted 12 lines

Quoting Saravana Kannan (2019-09-04 14:11:22)

Quoted 10 lines
drivers/of/property.c +10 -0
+       int ret = 0;
+       struct device_node *tmp_np = sup_np;
+
+       of_node_get(sup_np);
+       /*
+        * Find the device node that contains the supplier phandle.  It may be
+        * @sup_np or it may be an ancestor of @sup_np.
+        */
+       while (sup_np && !of_find_property(sup_np, "compatible", NULL))
+               sup_np = of_get_next_parent(sup_np);
Quoted 8 lines
Quoted 6 lines

I don't get this. This is assuming that drivers are only probed for device nodes that have a compatible string? What about drivers that make sub-devices for clk support that have drivers in drivers/clk/ that then attach at runtime later? This happens sometimes for MFDs that want to split the functionality across the driver tree to the respective subsystems.

For that, the link would not be there, correct?

The parent device (MFD) would have the links because that is the device node with the provider property like '#clock-cells'. The child clk device that's populated by the MFD would be the one actually providing the clk via a driver that may probe any time later, or never, depending on if the clk driver is configured as a module or not. I fail to see how this will work for these cases.

Is this logic there to find the parent of a regulator phandle and match that to some driver? It looks like it.

Quoted 20 lines
Quoted 11 lines
Quoted 11 lines
drivers/of/property.c +10 -0
+static int of_link_property(struct device *dev, struct device_node *con_np,
+                            const char *prop_name)
+{
+       struct device_node *phandle;
+       const struct supplier_bindings *s = bindings;
+       unsigned int i = 0;
+       bool matched = false;
+       int ret = 0;
+
+       /* Do not stop at first failed link, link all available suppliers. */
Quoted 8 lines

drivers/of/property.c of_fwnode_device_get_match_data(const struct fwnode_handle *fwnode, · line 1191

Quoted 6 lines
drivers/of/property.c +6 -0
+       while (!matched && s->parse_prop) {
+               while ((phandle = s->parse_prop(con_np, prop_name, i))) {
+                       matched = true;
+                       i++;
+                       if (of_link_to_phandle(dev, phandle) == -EAGAIN)
+                               ret = -EAGAIN;

And don't break?

Quoted 6 lines

There was comments before about how this is not needed. Frank asked that the comment be removed. And now you point it out again :)

Look at the comment a few lines up, we have to go through all of the suppliers.

Ok. The comment tells me what is happening but it misses the essential part which is _why_ we must make links to each supplier and return -EAGAIN.

#8 lore raw

Re: [PATCH v11 3/6] of: property: Add functional dependency link from DT bindings

Quoted 31 lines · Stephen Boyd

Quoting Greg Kroah-Hartman (2019-10-04 08:37:50)

Quoted 13 lines · Stephen Boyd
Quoted 12 lines

Quoting Saravana Kannan (2019-09-04 14:11:22)

Quoted 10 lines
drivers/of/property.c +10 -0
+       int ret = 0;
+       struct device_node *tmp_np = sup_np;
+
+       of_node_get(sup_np);
+       /*
+        * Find the device node that contains the supplier phandle.  It may be
+        * @sup_np or it may be an ancestor of @sup_np.
+        */
+       while (sup_np && !of_find_property(sup_np, "compatible", NULL))
+               sup_np = of_get_next_parent(sup_np);
Quoted 8 lines
Quoted 6 lines

I don't get this. This is assuming that drivers are only probed for device nodes that have a compatible string? What about drivers that make sub-devices for clk support that have drivers in drivers/clk/ that then attach at runtime later? This happens sometimes for MFDs that want to split the functionality across the driver tree to the respective subsystems.

For that, the link would not be there, correct?

The parent device (MFD) would have the links because that is the device node with the provider property like '#clock-cells'. The child clk device that's populated by the MFD would be the one actually providing the clk via a driver that may probe any time later, or never, depending on if the clk driver is configured as a module or not. I fail to see how this will work for these cases.

Quoted 2 lines

Is this logic there to find the parent of a regulator phandle and match that to some driver? It looks like it.

In the case of an MFD creating "fake" children devices, the parent MFD device's driver is responsible for handling the sync state callback. It'll get the sync_state callback after all the child devices' consumers have probed. The MFD driver will need to do the sync state clean up for the children devices or pass it on to the child devices' drivers (whatever makes sense for that specific MFD) by whatever means those specific drivers talk to each other (direct calls, registering callbacks, etc).

If they are real sub-devices, then they should really be captured in DT as child devices and then the child device's drivers will get the sync state callback directly.

Quoted 23 lines
Quoted 23 lines
Quoted 11 lines
Quoted 11 lines
drivers/of/property.c +10 -0
+static int of_link_property(struct device *dev, struct device_node *con_np,
+                            const char *prop_name)
+{
+       struct device_node *phandle;
+       const struct supplier_bindings *s = bindings;
+       unsigned int i = 0;
+       bool matched = false;
+       int ret = 0;
+
+       /* Do not stop at first failed link, link all available suppliers. */
Quoted 8 lines

drivers/of/property.c of_fwnode_device_get_match_data(const struct fwnode_handle *fwnode, · line 1191

Quoted 6 lines
drivers/of/property.c +6 -0
+       while (!matched && s->parse_prop) {
+               while ((phandle = s->parse_prop(con_np, prop_name, i))) {
+                       matched = true;
+                       i++;
+                       if (of_link_to_phandle(dev, phandle) == -EAGAIN)
+                               ret = -EAGAIN;

And don't break?

There was comments before about how this is not needed. Frank asked that the comment be removed. And now you point it out again :)

Quoted 7 lines
Quoted 3 lines

Look at the comment a few lines up, we have to go through all of the suppliers.

Ok. The comment tells me what is happening but it misses the essential part which is _why_ we must make links to each supplier and return -EAGAIN.

To be clear the -EAGAIN is only if any of the linking fails.

The reason was already discussion in the email thread [1] but I agree it needs to be documented.

I thought I had documented the _why_ in the documentation for fwnode.add_links(), but it's not there. I'll check to make sure I didn't capture it elsewhere and if not, I'll update fwnode.add_links documentation.

To copy-paste the discussion from the earlier thread:

"Actually, there is a point for this. Say Device-C depends on suppliers Device-S1 and Device-S2 and they are listed in DT in that order.

Say, S1 gets populated after late_initcall_sync but S2 is probes way before that. If I don't continue past a "failed linking" to S1 and also link up to S2, then S2 will get a sync_state() callback before C is probed. So I have to go through all possible suppliers and [link] as many as possible."

-Saravana

[1] - https://lore.kernel.org/lkml/CAGETcx-hCrUvY5whZBihueqqCxmF3oDjFybjmoo3JUu87iiiEw@mail.gmail.com/

#9 lore raw

Re: [PATCH v11 3/6] of: property: Add functional dependency link from DT bindings

Quoting Saravana Kannan (2019-10-08 11:57:49)

Quoted 44 lines · Stephen Boyd
Quoted 31 lines

Quoting Greg Kroah-Hartman (2019-10-04 08:37:50)

Quoted 13 lines · Stephen Boyd
Quoted 12 lines

Quoting Saravana Kannan (2019-09-04 14:11:22)

Quoted 10 lines
drivers/of/property.c +10 -0
+       int ret = 0;
+       struct device_node *tmp_np = sup_np;
+
+       of_node_get(sup_np);
+       /*
+        * Find the device node that contains the supplier phandle.  It may be
+        * @sup_np or it may be an ancestor of @sup_np.
+        */
+       while (sup_np && !of_find_property(sup_np, "compatible", NULL))
+               sup_np = of_get_next_parent(sup_np);
Quoted 8 lines
Quoted 6 lines

I don't get this. This is assuming that drivers are only probed for device nodes that have a compatible string? What about drivers that make sub-devices for clk support that have drivers in drivers/clk/ that then attach at runtime later? This happens sometimes for MFDs that want to split the functionality across the driver tree to the respective subsystems.

For that, the link would not be there, correct?

The parent device (MFD) would have the links because that is the device node with the provider property like '#clock-cells'. The child clk device that's populated by the MFD would be the one actually providing the clk via a driver that may probe any time later, or never, depending on if the clk driver is configured as a module or not. I fail to see how this will work for these cases.

Quoted 2 lines

Is this logic there to find the parent of a regulator phandle and match that to some driver? It looks like it.

In the case of an MFD creating "fake" children devices, the parent MFD device's driver is responsible for handling the sync state callback. It'll get the sync_state callback after all the child devices' consumers have probed. The MFD driver will need to do the sync state clean up for the children devices or pass it on to the child devices' drivers (whatever makes sense for that specific MFD) by whatever means those specific drivers talk to each other (direct calls, registering callbacks, etc).

Quoted 3 lines

If they are real sub-devices, then they should really be captured in DT as child devices and then the child device's drivers will get the sync state callback directly.

It seems sort of hand-wavy at the moment. Is the plan to actually implement this for MFDs that are doing these things? It's really hard to understand this patch series without any actual users.

From my perspective using driver probe as the signal that some resource like clks or regulators has been consumed and configured into the proper state is completely wrong. It makes a large assumption that driver probe is actually putting the device into some state that has taken over ownership of the device state by the time probe returns. That isn't always the case when you consider things like the display or GPU don't do much until their device is opened by userspace.

It would be better to involve the various kernel frameworks in this decision by having those frameworks intercept the acquisition of the resources they provide and track consumers to the point where we can be certain all consumers have requested and configured the resources they need to operate properly without something go wrong. Maybe we need drivers to indicate this to frameworks somehow so that we don't turn the regulator off for the screen when the screen driver probes but the GPU driver hasn't started drawing anything there because userspace isn't running yet?

I'm trying to take a step back and understand the bigger picture here. From what I can tell we're trying to answer the question "When have all the consumers of this resource put their constraints in place?" This is because we want to actively cleanup resources that have been left on or misconfigured by bootloader/firmware code but we can't be certain when to do that and if we should do that at all. Is that right?

#10 lore raw

PATCH v11 4/6 driver core: Add sync_state driver/bus callback

In reply to Saravana Kannan

This sync_state driver/bus callback is called once all the consumers of a supplier have probed successfully.

This allows the supplier device's driver/bus to sync the supplier device's state to the software state with the guarantee that all the consumers are actively managing the resources provided by the supplier device.

To maintain backwards compatibility and ease transition from existing frameworks and resource cleanup schemes, late_initcall_sync is the earliest when the sync_state callback might be called.

There is no upper bound on the time by which the sync_state callback has to be called. This is because if a consumer device never probes, the supplier has to maintain its resources in the state left by the bootloader. For example, if the bootloader leaves the display backlight at a fixed voltage and the backlight driver is never probed, you don't want the backlight to ever be turned off after boot up.

Also, when multiple devices are added after kernel init, some suppliers could be added before their consumer devices get added. In these instances, the supplier devices could get their sync_state callback called right after they probe because the consumers devices haven't had a chance to create device links to the suppliers.

To handle this correctly, this change also provides APIs to pause/resume sync state callbacks so that when multiple devices are added, their sync_state callback evaluation can be postponed to happen after all of them are added.

kbuild test robot reported missing documentation for device.state_synced

---
 drivers/base/core.c    | 72 ++++++++++++++++++++++++++++++++++++++++++
 include/linux/device.h | 24 ++++++++++++++
 2 files changed, 96 insertions(+)
drivers/base/core.c +72 -0
index b3896da73b3d..acbf0b1414ab 100644
@@ -46,6 +46,8 @@ early_param("sysfs.deprecated", sysfs_deprecated_setup);
 /* Device links support. */
 static LIST_HEAD(wait_for_suppliers);
 static DEFINE_MUTEX(wfs_lock);
+static LIST_HEAD(deferred_sync);
+static unsigned int defer_sync_state_count = 1;
 
 #ifdef CONFIG_SRCU
 static DEFINE_MUTEX(device_links_lock);
@@ -648,6 +650,69 @@ int device_links_check_suppliers(struct device *dev)
 	return ret;
 }
 
+static void __device_links_supplier_sync_state(struct device *dev)
+{
+	struct device_link *link;
+
+	if (dev->state_synced)
+		return;
+
+	list_for_each_entry(link, &dev->links.consumers, s_node) {
+		if (!(link->flags & DL_FLAG_MANAGED))
+			continue;
+		if (link->status != DL_STATE_ACTIVE)
+			return;
+	}
+
+	if (dev->bus->sync_state)
+		dev->bus->sync_state(dev);
+	else if (dev->driver && dev->driver->sync_state)
+		dev->driver->sync_state(dev);
+
+	dev->state_synced = true;
+}
+
+void device_links_supplier_sync_state_pause(void)
+{
+	device_links_write_lock();
+	defer_sync_state_count++;
+	device_links_write_unlock();
+}
+
+void device_links_supplier_sync_state_resume(void)
+{
+	struct device *dev, *tmp;
+
+	device_links_write_lock();
+	if (!defer_sync_state_count) {
+		WARN(true, "Unmatched sync_state pause/resume!");
+		goto out;
+	}
+	defer_sync_state_count--;
+	if (defer_sync_state_count)
+		goto out;
+
+	list_for_each_entry_safe(dev, tmp, &deferred_sync, links.defer_sync) {
+		__device_links_supplier_sync_state(dev);
+		list_del_init(&dev->links.defer_sync);
+	}
+out:
+	device_links_write_unlock();
+}
+
+static int sync_state_resume_initcall(void)
+{
+	device_links_supplier_sync_state_resume();
+	return 0;
+}
+late_initcall(sync_state_resume_initcall);
+
+static void __device_links_supplier_defer_sync(struct device *sup)
+{
+	if (list_empty(&sup->links.defer_sync))
+		list_add_tail(&sup->links.defer_sync, &deferred_sync);
+}
+
 /**
  * device_links_driver_bound - Update device links after probing its driver.
  * @dev: Device to update the links for.
@@ -692,6 +757,11 @@ void device_links_driver_bound(struct device *dev)
 
 		WARN_ON(link->status != DL_STATE_CONSUMER_PROBE);
 		WRITE_ONCE(link->status, DL_STATE_ACTIVE);
+
+		if (defer_sync_state_count)
+			__device_links_supplier_defer_sync(link->supplier);
+		else
+			__device_links_supplier_sync_state(link->supplier);
 	}
 
 	dev->links.status = DL_DEV_DRIVER_BOUND;
@@ -808,6 +878,7 @@ void device_links_driver_cleanup(struct device *dev)
 		WRITE_ONCE(link->status, DL_STATE_DORMANT);
 	}
 
+	list_del_init(&dev->links.defer_sync);
 	__device_links_no_driver(dev);
 
 	device_links_write_unlock();
@@ -1782,6 +1853,7 @@ void device_initialize(struct device *dev)
 	INIT_LIST_HEAD(&dev->links.consumers);
 	INIT_LIST_HEAD(&dev->links.suppliers);
 	INIT_LIST_HEAD(&dev->links.needs_suppliers);
+	INIT_LIST_HEAD(&dev->links.defer_sync);
 	dev->links.status = DL_DEV_NO_DRIVER;
 }
 EXPORT_SYMBOL_GPL(device_initialize);
include/linux/device.h +24 -0
index 968316bb3bd1..9f2f2e169f95 100644
@@ -81,6 +81,13 @@ extern void bus_remove_file(struct bus_type *, struct bus_attribute *);
  *		that generate uevents to add the environment variables.
  * @probe:	Called when a new device or driver add to this bus, and callback
  *		the specific driver's probe to initial the matched device.
+ * @sync_state:	Called to sync device state to software state after all the
+ *		state tracking consumers linked to this device (present at
+ *		the time of late_initcall) have successfully bound to a
+ *		driver. If the device has no consumers, this function will
+ *		be called at late_initcall_sync level. If the device has
+ *		consumers that are never bound to a driver, this function
+ *		will never get called until they do.
  * @remove:	Called when a device removed from this bus.
  * @shutdown:	Called at shut-down time to quiesce the device.
  *
@@ -124,6 +131,7 @@ struct bus_type {
 	int (*match)(struct device *dev, struct device_driver *drv);
 	int (*uevent)(struct device *dev, struct kobj_uevent_env *env);
 	int (*probe)(struct device *dev);
+	void (*sync_state)(struct device *dev);
 	int (*remove)(struct device *dev);
 	void (*shutdown)(struct device *dev);
 
@@ -341,6 +349,13 @@ enum probe_type {
  * @probe:	Called to query the existence of a specific device,
  *		whether this driver can work with it, and bind the driver
  *		to a specific device.
+ * @sync_state:	Called to sync device state to software state after all the
+ *		state tracking consumers linked to this device (present at
+ *		the time of late_initcall) have successfully bound to a
+ *		driver. If the device has no consumers, this function will
+ *		be called at late_initcall_sync level. If the device has
+ *		consumers that are never bound to a driver, this function
+ *		will never get called until they do.
  * @remove:	Called when the device is removed from the system to
  *		unbind a device from this driver.
  * @shutdown:	Called at shut-down time to quiesce the device.
@@ -380,6 +395,7 @@ struct device_driver {
 	const struct acpi_device_id	*acpi_match_table;
 
 	int (*probe) (struct device *dev);
+	void (*sync_state)(struct device *dev);
 	int (*remove) (struct device *dev);
 	void (*shutdown) (struct device *dev);
 	int (*suspend) (struct device *dev, pm_message_t state);
@@ -1137,12 +1153,14 @@ enum dl_dev_state {
  * @suppliers: List of links to supplier devices.
  * @consumers: List of links to consumer devices.
  * @needs_suppliers: Hook to global list of devices waiting for suppliers.
+ * @defer_sync: Hook to global list of devices that have deferred sync_state.
  * @status: Driver status information.
  */
 struct dev_links_info {
 	struct list_head suppliers;
 	struct list_head consumers;
 	struct list_head needs_suppliers;
+	struct list_head defer_sync;
 	enum dl_dev_state status;
 };
 
@@ -1218,6 +1236,9 @@ struct dev_links_info {
  * @offline:	Set after successful invocation of bus type's .offline().
  * @of_node_reused: Set if the device-tree node is shared with an ancestor
  *              device.
+ * @state_synced: The hardware state of this device has been synced to match
+ *		  the software state of this device by calling the driver/bus
+ *		  sync_state() callback.
  * @dma_coherent: this particular device is dma coherent, even if the
  *		architecture supports non-coherent devices.
  *
@@ -1314,6 +1335,7 @@ struct device {
 	bool			offline_disabled:1;
 	bool			offline:1;
 	bool			of_node_reused:1;
+	bool			state_synced:1;
 #if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_DEVICE) || \
     defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU) || \
     defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU_ALL)
@@ -1656,6 +1678,8 @@ struct device_link *device_link_add(struct device *consumer,
 				    struct device *supplier, u32 flags);
 void device_link_del(struct device_link *link);
 void device_link_remove(void *consumer, struct device *supplier);
+void device_links_supplier_sync_state_pause(void);
+void device_links_supplier_sync_state_resume(void);
 
 #ifdef CONFIG_DEVICE_NOTIFICATIONS
 extern void post_device_notification(struct watch_notification *n, u64 id);
Signature · 1 line
2.23.0.187.g17f5b7556c-goog
#11 lore raw

PATCH v11 5/6 of/platform: Pause/resume sync state during init and of_platform_populate()

In reply to Saravana Kannan

When all the top level devices are populated from DT during kernel init, the supplier devices could be added and probed before the consumer devices are added and linked to the suppliers. To avoid the sync_state() callback from being called prematurely, pause the sync_state() callbacks before populating the devices and resume them at late_initcall_sync().

Similarly, when children devices are populated from a module using of_platform_populate(), there could be supplier-consumer dependencies between the children devices that are populated. To avoid the same problem with sync_state() being called prematurely, pause and resume sync_state() callbacks across of_platform_populate().

---
 drivers/of/platform.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)
drivers/of/platform.c +12 -0
index b47a2292fe8e..d93891a05f60 100644
@@ -480,6 +480,7 @@ int of_platform_populate(struct device_node *root,
 	pr_debug("%s()\n", __func__);
 	pr_debug(" starting at: %pOF\n", root);
 
+	device_links_supplier_sync_state_pause();
 	for_each_child_of_node(root, child) {
 		rc = of_platform_bus_create(child, matches, lookup, parent, true);
 		if (rc) {
@@ -487,6 +488,8 @@ int of_platform_populate(struct device_node *root,
 			break;
 		}
 	}
+	device_links_supplier_sync_state_resume();
+
 	of_node_set_flag(root, OF_POPULATED_BUS);
 
 	of_node_put(root);
@@ -518,6 +521,7 @@ static int __init of_platform_default_populate_init(void)
 	if (!of_have_populated_dt())
 		return -ENODEV;
 
+	device_links_supplier_sync_state_pause();
 	/*
 	 * Handle certain compatibles explicitly, since we don't want to create
 	 * platform_devices for every node in /reserved-memory with a
@@ -538,6 +542,14 @@ static int __init of_platform_default_populate_init(void)
 	return 0;
 }
 arch_initcall_sync(of_platform_default_populate_init);
+
+static int __init of_platform_sync_state_init(void)
+{
+	if (of_have_populated_dt())
+		device_links_supplier_sync_state_resume();
+	return 0;
+}
+late_initcall_sync(of_platform_sync_state_init);
 #endif
 
 int of_platform_device_destroy(struct device *dev, void *data)
Signature · 1 line
2.23.0.187.g17f5b7556c-goog
#12 lore raw

PATCH v11 6/6 of: property: Create device links for all child-supplier depencencies

In reply to Saravana Kannan

A parent device can have child devices that it adds when it probes. But this probing of the parent device can happen way after kernel init is done -- for example, when the parent device's driver is loaded as a module.

In such cases, if the child devices depend on a supplier in the system, we need to make sure the supplier gets the sync_state() callback only after these child devices are added and probed.

To achieve this, when creating device links for a device by looking at its DT node, don't just look at DT references at the top node level. Look at DT references in all the descendant nodes too and create device links from the ancestor device to all these supplier devices.

This way, when the parent device probes and adds child devices, the child devices can then create their own device links to the suppliers and further delay the supplier's sync_state() callback to after the child devices are probed.

Example:
In this illustration, -> denotes DT references and indentation represents child status.

Device node A

	Device node B -> D
	Device node C -> B, D

Device node D

Assume all these devices have their drivers loaded as modules.

Without this patch, this is the sequence of events:
1. D is added.
2. A is added.
3. Device D probes.
4. Device D gets its sync_state() callback.
5. Device B and C might malfunction because their resources got altered/turned off before they can make active requests for them.

With this patch, this is the sequence of events:
1. D is added.
2. A is added and creates device links to D.
3. Device link from A to B is not added because A is a parent of B.
4. Device D probes.
5. Device D does not get it's sync_state() callback because consumer A hasn't probed yet.
5. Device A probes.
5. a. Devices B and C are added.
5. b. Device links from B and C to D are added.
5. c. Device A's probe completes.
6. Device D does not get it's sync_state() callback because consumer A has probed but consumers B and C haven't probed yet.
7. Device B and C probe.
8. Device D gets it's sync_state() callback because all its consumers have probed.
9. None of the devices malfunction.

---
 drivers/of/property.c | 4 ++++
 1 file changed, 4 insertions(+)
drivers/of/property.c +4 -0
index 23b5ee5b0570..923d6f88a99c 100644
@@ -1207,6 +1207,10 @@ static int __of_link_to_suppliers(struct device *dev,
 		if (of_link_property(dev, con_np, p->name))
 			ret = -EAGAIN;
 
+	for_each_child_of_node(con_np, child)
+		if (__of_link_to_suppliers(dev, child))
+			ret = -EAGAIN;
+
 	return ret;
 }
 
Signature · 1 line
2.23.0.187.g17f5b7556c-goog
#13 lore raw
In reply to Saravana Kannan

Quoting Saravana Kannan (2019-09-04 14:11:19)

Quoted 4 lines

v10->v11:
- Dropped 6/7 and 7/7 from previous series that tried to handle cycles in DT dependencies. We can solve it later when we actually hit a real world issue in DT.

Quoted 8 lines

- Added a new 1/7 that shifts the numbering for the rest of the patches
- 1/7 adds a way to look up a device from a fwnode so that this series can work across bus and firmware types
- 3/7 removed references to platform_device from of/property.c
- 4/7 Minor variable rename
- 4/7 Defer sync_state() be default at driver core level and resume at late_initcall_sync(). That way, we don't depend on any specific bus types having to pause/resume sync_state() till late_initcall_sync()

Please reverse this list so that most recent series changes come first and we don't have to scroll through all the history to get to what has changed recently.

Also, please Cc me on future changes and I would suggest involving any subsystem maintainers that this code is optimizing for. Looks like Mark Brown and Georgi Djakov should be included in this series for awareness (but it's at v11 already!)

#14 lore raw
In reply to Saravana Kannan
Quoted 40 lines · Saravana Kannan

Add device-links to track functional dependencies between devices after they are created (but before they are probed) by looking at their common DT bindings like clocks, interconnects, etc.

Having functional dependencies automatically added before the devices are probed, provides the following benefits:

- Optimizes device probe order and avoids the useless work of attempting probes of devices that will not probe successfully (because their suppliers aren't present or haven't probed yet).

For example, in a commonly available mobile SoC, registering just one consumer device's driver at an initcall level earlier than the supplier device's driver causes 11 failed probe attempts before the consumer device probes successfully. This was with a kernel with all the drivers statically compiled in. This problem gets a lot worse if all the drivers are loaded as modules without direct symbol dependencies.

- Supplier devices like clock providers, interconnect providers, etc need to keep the resources they provide active and at a particular state(s) during boot up even if their current set of consumers don't request the resource to be active. This is because the rest of the consumers might not have probed yet and turning off the resource before all the consumers have probed could lead to a hang or undesired user experience.

Some frameworks (Eg: regulator) handle this today by turning off "unused" resources at late_initcall_sync and hoping all the devices have probed by then. This is not a valid assumption for systems with loadable modules. Other frameworks (Eg: clock) just don't handle this due to the lack of a clear signal for when they can turn off resources. This leads to downstream hacks to handle cases like this that can easily be solved in the upstream kernel.

By linking devices before they are probed, we give suppliers a clear count of the number of dependent consumers. Once all of the consumers are active, the suppliers can turn off the unused resources without making assumptions about the number of consumers.

Quoted 4 lines

By default we just add device-links to track "driver presence" (probe succeeded) of the supplier device. If any other functionality provided by device-links are needed, it is left to the consumer/supplier devices to change the link when they probe.

Hearing no real complaints again, I've now queued these up in my tree.

thanks,

greg k-h