Solve postboot supplier cleanup and optimize probe ordering
Subsystems (3)
v7 cover Solve postboot supplier cleanup and optimize probe ordering
Add device-links to track functional dependencies between devices after they are created (but before they are probed) by looking at their common DT bindings like clocks, interconnects, etc.
Having functional dependencies automatically added before the devices are probed, provides the following benefits:
- Optimizes device probe order and avoids the useless work of attempting probes of devices that will not probe successfully (because their suppliers aren't present or haven't probed yet).
For example, in a commonly available mobile SoC, registering just one consumer device's driver at an initcall level earlier than the supplier device's driver causes 11 failed probe attempts before the consumer device probes successfully. This was with a kernel with all the drivers statically compiled in. This problem gets a lot worse if all the drivers are loaded as modules without direct symbol dependencies.
- Supplier devices like clock providers, interconnect providers, etc need to keep the resources they provide active and at a particular state(s) during boot up even if their current set of consumers don't request the resource to be active. This is because the rest of the consumers might not have probed yet and turning off the resource before all the consumers have probed could lead to a hang or undesired user experience.
Some frameworks (Eg: regulator) handle this today by turning off "unused" resources at late_initcall_sync and hoping all the devices have probed by then. This is not a valid assumption for systems with loadable modules. Other frameworks (Eg: clock) just don't handle this due to the lack of a clear signal for when they can turn off resources. This leads to downstream hacks to handle cases like this that can easily be solved in the upstream kernel.
By linking devices before they are probed, we give suppliers a clear count of the number of dependent consumers. Once all of the consumers are active, the suppliers can turn off the unused resources without making assumptions about the number of consumers.
By default we just add device-links to track "driver presence" (probe succeeded) of the supplier device. If any other functionality provided by device-links are needed, it is left to the consumer/supplier devices to change the link when they probe.
v1 -> v2:
- Drop patch to speed up of_find_device_by_node()
- Drop depends-on property and use existing bindings
v2 -> v3:
- Refactor the code to have driver core initiate the linking of devs
- Have driver core link consumers to supplier before it's probed
- Add support for drivers to edit the device links before probing
v3 -> v4:
- Tested edit_links() on system with cyclic dependency. Works.
- Added some checks to make sure device link isn't attempted from parent device node to child device node.
- Added way to pause/resume sync_state callbacks across of_platform_populate().
- Recursively parse DT node to create device links from parent to suppliers of parent and all child nodes.
v4 -> v5:
- Fixed copy-pasta bugs with linked list handling
- Walk up the phandle reference till I find an actual device (needed for regulators to work)
- Added support for linking devices from regulator DT bindings
- Tested the whole series again to make sure cyclic dependencies are broken with edit_links() and regulator links are created properly.
v5 -> v6:
- Split, squashed and reordered some of the patches.
- Refactored the device linking code to follow the same code pattern for any property.
v6 -> v7:
- No functional changes.
- Renamed i to index
- Added comment to clarify not having to check property name for every index
- Added "matched" variable to clarify code. No functional change.
- Added comments to include/linux/device.h for add_links()
I've also not updated this patch series to handle the new patch [1] from Rafael. Will do that once this patch series is close to being Acked.
[1] - https://lore.kernel.org/lkml/3121545.4lOhFoIcdQ@kreacher/
-Saravana
Saravana Kannan (7):
driver core: Add support for linking devices during device addition driver core: Add edit_links() callback for drivers of/platform: Add functional dependency link from DT bindings driver core: Add sync_state driver/bus callback of/platform: Pause/resume sync state during init and
of_platform_populate()
of/platform: Create device links for all child-supplier depencencies of/platform: Don't create device links for default busses
.../admin-guide/kernel-parameters.txt | 5 + drivers/base/core.c | 168 ++++++++++++++++ drivers/base/dd.c | 29 +++ drivers/of/platform.c | 189 ++++++++++++++++++ include/linux/device.h | 55 +++++ 5 files changed, 446 insertions(+)
Signature · 1 line
2.22.0.709.g102302147b-goog
PATCH v7 1/7 driver core: Add support for linking devices during device addition
When devices are added, the bus might want to create device links to track functional dependencies between supplier and consumer devices. This tracking of supplier-consumer relationship allows optimizing device probe order and tracking whether all consumers of a supplier are active. The add_links bus callback is added to support this.
However, when consumer devices are added, they might not have a supplier device to link to despite needing mandatory resources/functionality from one or more suppliers. A waiting_for_suppliers list is created to track such consumers and retry linking them when new devices get added.
--- drivers/base/core.c | 83 ++++++++++++++++++++++++++++++++++++++++++ include/linux/device.h | 14 +++++++ 2 files changed, 97 insertions(+)
drivers/base/core.c +83 -0
@@ -44,6 +44,8 @@ early_param("sysfs.deprecated", sysfs_deprecated_setup);#endif /* Device links support. */ +static LIST_HEAD (wait_for_suppliers); +static DEFINE_MUTEX (wfs_lock);#ifdef CONFIG_SRCU static DEFINE_MUTEX (device_links_lock);
@@ -401,6 +403,51 @@ struct device_link *device_link_add(struct device *consumer, }EXPORT_SYMBOL_GPL (device_link_add); +/** +* device_link_wait_for_supplier - Mark device as waiting for supplier +* @consumer: Consumer device +* +* Marks the consumer device as waiting for suppliers to become available. The +* consumer device will never be probed until it's unmarked as waiting for +* suppliers. The caller is responsible for adding the link to the supplier +* once the supplier device is present. +* +* This function is NOT meant to be called from the probe function of the +* consumer but rather from code that creates/adds the consumer device. +*/ +static void device_link_wait_for_supplier (struct device *consumer) +{ +mutex_lock (&wfs_lock); +list_add_tail (&consumer->links.needs_suppliers, &wait_for_suppliers); +mutex_unlock (&wfs_lock); +} + +/** +* device_link_check_waiting_consumers - Try to unmark waiting consumers +* +* Loops through all consumers waiting on suppliers and tries to add all their +* supplier links. If that succeeds, the consumer device is unmarked as waiting +* for suppliers. Otherwise, they are left marked as waiting on suppliers, +* +* The add_links bus callback is expected to return 0 if it has found and added +* all the supplier links for the consumer device. It should return an error if +* it isn't able to do so. +* +* The caller of device_link_wait_for_supplier() is expected to call this once +* it's aware of potential suppliers becoming available. +*/ +static void device_link_check_waiting_consumers (void ) +{ +struct device *dev, *tmp; + +mutex_lock (&wfs_lock); +list_for_each_entry_safe (dev, tmp, &wait_for_suppliers, + links.needs_suppliers) +if (!dev->bus->add_links (dev)) +list_del_init (&dev->links.needs_suppliers); +mutex_unlock (&wfs_lock); +} +static void device_link_free (struct device_link *link) {while (refcount_dec_not_one (&link->rpm_active))
@@ -535,6 +582,19 @@ int device_links_check_suppliers(struct device *dev)struct device_link *link;int ret =0 ; +/* +* If a device is waiting for one or more suppliers (in +* wait_for_suppliers list), it is not ready to probe yet. So just +* return -EPROBE_DEFER without having to check the links with existing +* suppliers. +*/ +mutex_lock (&wfs_lock); +if (!list_empty (&dev->links.needs_suppliers)) { +mutex_unlock (&wfs_lock); +return -EPROBE_DEFER ; + } +mutex_unlock (&wfs_lock); +device_links_write_lock ();list_for_each_entry (link, &dev->links.suppliers, c_node) {
@@ -812,6 +872,10 @@ static void device_links_purge(struct device *dev) {struct device_link *link, *ln; +mutex_lock (&wfs_lock); +list_del (&dev->links.needs_suppliers); +mutex_unlock (&wfs_lock); +/* * Delete all of the remaining links from this device to any other * devices (either consumers or suppliers).
@@ -1673,6 +1737,7 @@ void device_initialize(struct device *dev)#endif INIT_LIST_HEAD (&dev->links.consumers);INIT_LIST_HEAD (&dev->links.suppliers); +INIT_LIST_HEAD (&dev->links.needs_suppliers); dev->links.status =DL_DEV_NO_DRIVER ; }EXPORT_SYMBOL_GPL (device_initialize);
@@ -2108,6 +2173,24 @@ int device_add(struct device *dev) BUS_NOTIFY_ADD_DEVICE, dev);kobject_uevent (&dev->kobj,KOBJ_ADD ); + +/* +* Check if any of the other devices (consumers) have been waiting for +* this device (supplier) to be added so that they can create a device +* link to it. +* +* This needs to happen after device_pm_add() because device_link_add() +* requires the supplier be registered before it's called. +* +* But this also needs to happe before bus_probe_device() to make sure +* waiting consumers can link to it before the driver is bound to the +* device and the driver sync_state callback is called for this device. +*/ +device_link_check_waiting_consumers (); + +if (dev->bus && dev->bus->add_links && dev->bus->add_links (dev)) +device_link_wait_for_supplier (dev); +bus_probe_device (dev);if (parent) klist_add_tail(&dev->p->knode_parent,
include/linux/device.h +14 -0
@@ -78,6 +78,17 @@ extern void bus_remove_file(struct bus_type *, struct bus_attribute *);* -EPROBE_DEFER it will queue the device for deferred probing. * @uevent: Called when a device is added, removed, or a few other things * that generate uevents to add the environment variables. +* @add_links: Called, perhaps multiple times per device, after a device is +* added to this bus. The function is expected to create device +* links to all the suppliers of the input device that are +* available at the time this function is called. As in, the +* function should NOT stop at the first failed device link if +* other unlinked supplier devices are present in the system. +* +* Return 0 if device links have been successfully created to all +* the suppliers of this device. Return an error if some of the +* suppliers are not yet available and this function needs to be +* reattempted in the future. * @probe: Called when a new device or driver add to this bus, and callback * the specific driver's probe to initial the matched device. * @remove: Called when a device removed from this bus.
@@ -122,6 +133,7 @@ struct bus_type {int (*match)(struct device *dev,struct device_driver *drv);int (*uevent)(struct device *dev,struct kobj_uevent_env *env); +int (*add_links)(struct device *dev);int (*probe)(struct device *dev);int (*remove)(struct device *dev);void (*shutdown)(struct device *dev);
@@ -893,11 +905,13 @@ enum dl_dev_state {* struct dev_links_info - Device data related to device links. * @suppliers: List of links to supplier devices. * @consumers: List of links to consumer devices. +* @needs_suppliers: Hook to global list of devices waiting for suppliers. * @status: Driver status information. */ struct dev_links_info {struct list_head suppliers;struct list_head consumers; +struct list_head needs_suppliers;enum dl_dev_state status; };
Signature · 1 line
2.22.0.709.g102302147b-goog
Re: [PATCH v7 1/7] driver core: Add support for linking devices during device addition
Quoted 10 lines
Date: Tue, 23 Jul 2019 17:10:54 -0700
Subject: [PATCH v7 1/7] driver core: Add support for linking devices during
device addition
From: Saravana Kannan <[email protected]>When devices are added, the bus might want to create device links to track functional dependencies between supplier and consumer devices. This tracking of supplier-consumer relationship allows optimizing device probe order and tracking whether all consumers of a supplier are active. The
add_linksbus callback is added to support this.
Change above to:
When devices are added, the bus may create device links to track which suppliers a consumer device depends upon. This
tracking of supplier-consumer relationship may be used to defer probing the driver of a consumer device before the driver(s) for its supplier device(s) are probed. It may also be used by a supplier driver to determine if all of its consumers have been successfully probed.
The add_links bus callback is added to create the supplier device links
Quoted 5 lines
However, when consumer devices are added, they might not have a supplier device to link to despite needing mandatory resources/functionality from one or more suppliers. A
waiting_for_supplierslist is created to track such consumers and retry linking them when new devices get added.
Change above to:
If a supplier device has not yet been created when the consumer device attempts to link it, the consumer device is added to the wait_for_suppliers list. When supplier devices are created, the supplier device link will be added to the relevant consumer devices on the wait_for_suppliers list.
Quoted 21 lines
Signed-off-by Saravana Kannan <[email protected]>--- drivers/base/core.c | 83 ++++++++++++++++++++++++++++++++++++++++++ include/linux/device.h | 14 +++++++ 2 files changed, 97 insertions(+)drivers/base/core.c +2
-0@@ -44,6 +44,8 @@ early_param("sysfs.deprecated", sysfs_deprecated_setup);#endif /* Device links support. */ +static LIST_HEAD (wait_for_suppliers); +static DEFINE_MUTEX (wfs_lock);#ifdef CONFIG_SRCU static DEFINE_MUTEX (device_links_lock);
drivers/base/core.c struct device_link *device_link_add(struct device *consumer, · line 408
Quoted 6 lines
drivers/base/core.c +2
-0@@ -401,6 +403,51 @@ struct device_link *device_link_add(struct device *consumer, }EXPORT_SYMBOL_GPL (device_link_add); +/** +* device_link_wait_for_supplier - Mark device as waiting for supplier
* device_link_wait_for_supplier - Add device to wait_for_suppliers list
Quoted 22 lines
drivers/base/core.c +20
-0+* @consumer: Consumer device +* +* Marks the consumer device as waiting for suppliers to become available. The +* consumer device will never be probed until it's unmarked as waiting for +* suppliers. The caller is responsible for adding the link to the supplier +* once the supplier device is present. +* +* This function is NOT meant to be called from the probe function of the +* consumer but rather from code that creates/adds the consumer device. +*/ +static void device_link_wait_for_supplier (struct device *consumer) +{ +mutex_lock (&wfs_lock); +list_add_tail (&consumer->links.needs_suppliers, &wait_for_suppliers); +mutex_unlock (&wfs_lock); +} + +/** +* device_link_check_waiting_consumers - Try to remove from supplier wait list +*
drivers/base/core.c struct device_link *device_link_add(struct device *consumer, · line 437
Quoted 10 lines
drivers/base/core.c +10
-0+* Loops through all consumers waiting on suppliers and tries to add all their +* supplier links. If that succeeds, the consumer device is unmarked as waiting +* for suppliers. Otherwise, they are left marked as waiting on suppliers, +* +* The add_links bus callback is expected to return 0 if it has found and added +* all the supplier links for the consumer device. It should return an error if +* it isn't able to do so. +* +* The caller of device_link_wait_for_supplier() is expected to call this once +* it's aware of potential suppliers becoming available.
Change above comment to:
* device_link_add_supplier_links - add links from consumer devices to
* supplier devices, leaving any consumer
* with inactive suppliers on the
* wait_for_suppliers list
* Scan all consumer devices in the devicetree. For any supplier device that
* is not already linked to the consumer device, add the supplier to the
* consumer device's device links.
*
* If all of a consumer device's suppliers are available then the consumer
* is removed from the wait_for_suppliers list (if previously on the list).
* Otherwise the consumer is added to the wait_for_suppliers list (if not
* already on the list).
* The add_links bus callback must return 0 if it has found and added all
* the supplier links for the consumer device. It must return an error if
* it is not able to do so.
*
* The caller of device_link_wait_for_supplier() is expected to call this once
* it is aware of potential suppliers becoming available.
drivers/base/core.c struct device_link *device_link_add(struct device *consumer, · line 442
Quoted 2 lines
drivers/base/core.c +2
-0+*/ +static void device_link_check_waiting_consumers (void )
Function name is misleading and hides side effects.
I have not come up with a name that does not hide side effects, but a better name would be:
device_link_add_supplier_links()
drivers/base/core.c struct device_link *device_link_add(struct device *consumer, · line 447
Quoted 8 lines
drivers/base/core.c +8
-0+{ +struct device *dev, *tmp; + +mutex_lock (&wfs_lock); +list_for_each_entry_safe (dev, tmp, &wait_for_suppliers, + links.needs_suppliers) +if (!dev->bus->add_links (dev)) +list_del_init (&dev->links.needs_suppliers);
Empties dev->links.needs_suppliers, but does not remove dev from wait_for_suppliers list. Where does that happen?
Quoted 7 lines
drivers/base/core.c +3
-0+mutex_unlock (&wfs_lock); +} +static void
device_link_free(struct device_link *link){drivers/base/core.c +0
-0while (refcount_dec_not_one (&link->rpm_active))
drivers/base/core.c int device_links_check_suppliers(struct device *dev) · line 590
Quoted 9 lines
drivers/base/core.c +6
-0@@ -535,6 +582,19 @@ int device_links_check_suppliers(struct device *dev)struct device_link *link;int ret =0 ; +/* +* If a device is waiting for one or more suppliers (in +* wait_for_suppliers list), it is not ready to probe yet. So just +* return -EPROBE_DEFER without having to check the links with existing +* suppliers. +*/
Change comment to:
/* * Device waiting for supplier to become available is not allowed * to probe */
drivers/base/core.c int device_links_check_suppliers(struct device *dev) · line 598
Quoted 8 lines
drivers/base/core.c +7
-0+mutex_lock (&wfs_lock); +if (!list_empty (&dev->links.needs_suppliers)) { +mutex_unlock (&wfs_lock); +return -EPROBE_DEFER ; + } +mutex_unlock (&wfs_lock); +device_links_write_lock ();
Update Documentation/driver-api/device_link.rst to reflect the check of &dev->links.needs_suppliers in device_links_check_suppliers().
Quoted 29 lines
drivers/base/core.c +8
-0list_for_each_entry (link, &dev->links.suppliers, c_node) {@@ -812,6 +872,10 @@ static void device_links_purge(struct device *dev) {struct device_link *link, *ln; +mutex_lock (&wfs_lock); +list_del (&dev->links.needs_suppliers); +mutex_unlock (&wfs_lock); +/* * Delete all of the remaining links from this device to any other * devices (either consumers or suppliers). @@ -1673,6 +1737,7 @@ void device_initialize(struct device *dev)#endif INIT_LIST_HEAD (&dev->links.consumers);INIT_LIST_HEAD (&dev->links.suppliers); +INIT_LIST_HEAD (&dev->links.needs_suppliers); dev->links.status =DL_DEV_NO_DRIVER ; }EXPORT_SYMBOL_GPL (device_initialize);@@ -2108,6 +2173,24 @@ int device_add(struct device *dev) BUS_NOTIFY_ADD_DEVICE, dev);kobject_uevent (&dev->kobj,KOBJ_ADD ); + +/* +* Check if any of the other devices (consumers) have been waiting for
drivers/base/core.c int device_add(struct device *dev) · line 2189
Quoted 10 lines
drivers/base/core.c +10
-0+* this device (supplier) to be added so that they can create a device +* link to it. +* +* This needs to happen after device_pm_add() because device_link_add() +* requires the supplier be registered before it's called. +* +* But this also needs to happe before bus_probe_device() to make sure +* waiting consumers can link to it before the driver is bound to the +* device and the driver sync_state callback is called for this device. +*/
/*
* Add links to dev from any dependent consumer that has dev on it's
* list of needed suppliers (links.needs_suppliers). Device_pm_add()
* must have previously registered dev to allow the links to be added. *
* The consumer links must be created before dev is probed because the
* sync_state callback for dev will use the consumer links. */
Quoted 13 lines
drivers/base/core.c +5
-0+device_link_check_waiting_consumers (); + +if (dev->bus && dev->bus->add_links && dev->bus->add_links (dev)) +device_link_wait_for_supplier (dev); +bus_probe_device (dev);if (parent)drivers/base/core.c +0
-0klist_add_tail(&dev->p->knode_parent,include/linux/device.h +0
-0
include/linux/device.h extern void bus_remove_file(struct bus_type *, struct bus_attribute *); · line 87
Quoted 10 lines
include/linux/device.h +6
-0@@ -78,6 +78,17 @@ extern void bus_remove_file(struct bus_type *, struct bus_attribute *);* -EPROBE_DEFER it will queue the device for deferred probing. * @uevent: Called when a device is added, removed, or a few other things * that generate uevents to add the environment variables. +* @add_links: Called, perhaps multiple times per device, after a device is +* added to this bus. The function is expected to create device +* links to all the suppliers of the input device that are +* available at the time this function is called. As in, the +* function should NOT stop at the first failed device link if +* other unlinked supplier devices are present in the system.
* @add_links: Called after a device is added to this bus. The function is
* expected to create device links to all the suppliers of the
* device that are available at the time this function is called.
* The function must NOT stop at the first failed device link if
* other unlinked supplier devices are present in the system.
* If some suppliers are not yet available, this function will be
* called again when the suppliers become available.
but add_links() not needed, so moving this comment to of_link_to_suppliers()
include/linux/device.h extern void bus_remove_file(struct bus_type *, struct bus_attribute *); · line 91
Quoted 5 lines
include/linux/device.h +5
-0+* +* Return 0 if device links have been successfully created to all +* the suppliers of this device. Return an error if some of the +* suppliers are not yet available and this function needs to be +* reattempted in the future.
*
* Return 0 if device links have been successfully created to all
* the suppliers of this device. Return an error if some of the
* suppliers are not yet available.
include/linux/device.h struct bus_type { · line 138
Quoted 10 lines
include/linux/device.h +0
-0* @probe: Called when a new device or driver add to this bus, and callback * the specific driver's probe to initial the matched device. * @remove: Called when a device removed from this bus.
include/linux/device.h +1
-0@@ -122,6 +133,7 @@ struct bus_type {int (*match)(struct device *dev,struct device_driver *drv);int (*uevent)(struct device *dev,struct kobj_uevent_env *env); +int (*add_links)(struct device *dev);
^^^^^^^^^ add_supplier ???
^^^^^^^^^ add_suppliers ??? ^^^^^^^^^ link_suppliers ???
^^^^^^^^^ add_supplier_dependency ???
^^^^^^^^^ add_supplier_dependencies ???add_links() not needed
include/linux/device.h enum dl_dev_state { · line 909
Quoted 13 lines
include/linux/device.h +0
-0int (*probe)(struct device *dev);include/linux/device.h +1
-0int (*remove)(struct device *dev);void (*shutdown)(struct device *dev);@@ -893,11 +905,13 @@ enum dl_dev_state {* struct dev_links_info - Device data related to device links. * @suppliers: List of links to supplier devices. * @consumers: List of links to consumer devices. +* @needs_suppliers: Hook to global list of devices waiting for suppliers.
* @needs_suppliers: List of devices deferring probe until supplier drivers
* are successfully probed.
Quoted 13 lines
include/linux/device.h +1
-0* @status: Driver status information. */ struct dev_links_info {struct list_head suppliers;struct list_head consumers; +struct list_head needs_suppliers;enum dl_dev_state status; };Signature · 2 lines
-- 2.22.0.709.g102302147b-goog
Re: [PATCH v7 1/7] driver core: Add support for linking devices during device addition
Quoted 28 lines · Frank Rowand
Quoted 11 lines
Date: Tue, 23 Jul 2019 17:10:54 -0700
Subject: [PATCH v7 1/7] driver core: Add support for linking devices during
device addition
From: Saravana Kannan <[email protected]>When devices are added, the bus might want to create device links to track functional dependencies between supplier and consumer devices. This tracking of supplier-consumer relationship allows optimizing device probe order and tracking whether all consumers of a supplier are active. The
add_linksbus callback is added to support this.Change above to:
When devices are added, the bus may create device links to track which suppliers a consumer device depends upon. This
tracking of supplier-consumer relationship may be used to defer probing the driver of a consumer device before the driver(s) for its supplier device(s) are probed. It may also be used by a supplier driver to determine if all of its consumers have been successfully probed.
Theadd_linksbus callback is added to create the supplier device linksQuoted 5 lines
However, when consumer devices are added, they might not have a supplier device to link to despite needing mandatory resources/functionality from one or more suppliers. A
waiting_for_supplierslist is created to track such consumers and retry linking them when new devices get added.
Quoted 7 lines
Change above to:
If a supplier device has not yet been created when the consumer device attempts to link it, the consumer device is added to the
wait_for_supplierslist. When supplier devices are created, the supplier device link will be added to the relevant consumer devices on thewait_for_supplierslist.
I'll take these commit text suggestions if we decide to revert the entire series at the end of this review.
Quoted 24 lines
Quoted 24 lines
Signed-off-by Saravana Kannan <[email protected]>--- drivers/base/core.c | 83 ++++++++++++++++++++++++++++++++++++++++++ include/linux/device.h | 14 +++++++ 2 files changed, 97 insertions(+)drivers/base/core.c +2
-0@@ -44,6 +44,8 @@ early_param("sysfs.deprecated", sysfs_deprecated_setup);#endif /* Device links support. */ +static LIST_HEAD (wait_for_suppliers); +static DEFINE_MUTEX (wfs_lock);#ifdef CONFIG_SRCU static DEFINE_MUTEX (device_links_lock);@@ -401,6 +403,51 @@ struct device_link *device_link_add(struct device *consumer, }EXPORT_SYMBOL_GPL (device_link_add);
Quoted 5 lines
drivers/base/core.c struct device_link *device_link_add(struct device *consumer, · line 407
Quoted 3 lines
drivers/base/core.c +2
-0+/** +* device_link_wait_for_supplier - Mark device as waiting for supplier *
device_link_wait_for_supplier- Add device towait_for_supplierslist
I intentionally chose "Mark device..." because that's a better description of the semantics of the function instead of trying to describe the implementation. Whether I'm using a linked list or some other data structure should not be the one line documentation of a function. Unless the function is explicitly about operating on that specific data structure.
Quoted 35 lines
Quoted 24 lines
drivers/base/core.c +20
-0+* @consumer: Consumer device +* +* Marks the consumer device as waiting for suppliers to become available. The +* consumer device will never be probed until it's unmarked as waiting for +* suppliers. The caller is responsible for adding the link to the supplier +* once the supplier device is present. +* +* This function is NOT meant to be called from the probe function of the +* consumer but rather from code that creates/adds the consumer device. +*/ +static void device_link_wait_for_supplier (struct device *consumer) +{ +mutex_lock (&wfs_lock); +list_add_tail (&consumer->links.needs_suppliers, &wait_for_suppliers); +mutex_unlock (&wfs_lock); +} + +/** +* device_link_check_waiting_consumers - Try to remove from supplier wait list +* drivers/base/core.c struct device_link *device_link_add(struct device *consumer, · line 437
Quoted 10 lines
drivers/base/core.c +10
-0+* Loops through all consumers waiting on suppliers and tries to add all their +* supplier links. If that succeeds, the consumer device is unmarked as waiting +* for suppliers. Otherwise, they are left marked as waiting on suppliers, +* +* The add_links bus callback is expected to return 0 if it has found and added +* all the supplier links for the consumer device. It should return an error if +* it isn't able to do so. +* +* The caller of device_link_wait_for_supplier() is expected to call this once +* it's aware of potential suppliers becoming available.
Quoted 6 lines
Change above comment to:
*
device_link_add_supplier_links- add links from consumer devices to
* supplier devices, leaving any consumer
* with inactive suppliers on the
*wait_for_supplierslist
I didn't know that the first one line comment could span multiple lines. Good to know.
Quoted 1 line
* Scan all consumer devices in the devicetree.
This function doesn't have anything to do with devicetree. I've intentionally kept all OF related parts out of the driver/core because I hope that other busses can start using this feature too. So I can't take this bit.
Quoted 8 lines
For any supplier device that
* is not already linked to the consumer device, add the supplier to the
* consumer device's device links.
*
* If all of a consumer device's suppliers are available then the consumer
* is removed from thewait_for_supplierslist (if previously on the list).
* Otherwise the consumer is added to thewait_for_supplierslist (if not
* already on the list).
Honestly, I don't think this is any better than what I already have.
Quoted 12 lines
* The
add_linksbus callback must return 0 if it has found and added all
* the supplier links for the consumer device. It must return an error if
* it is not able to do so.
*
* The caller ofdevice_link_wait_for_supplier()is expected to call this once
* it is aware of potential suppliers becoming available.drivers/base/core.c struct device_link *device_link_add(struct device *consumer, · line 442
Quoted 2 lines
drivers/base/core.c +2
-0+*/ +static void device_link_check_waiting_consumers (void )
Quoted 6 lines
Function name is misleading and hides side effects.
I have not come up with a name that does not hide side effects, but a better name would be:
device_link_add_supplier_links()
I kinda agree that it could afford a better name. The current name is too similar to device_links_check_suppliers() and I never liked that.
Maybe device_link_add_missing_suppliers()?
I don't think we need "links" repeated twice in the function name. With this suggestion, what side effect is hidden in your opinion? That the fully linked consumer is removed from the "waiting for suppliers" list?
Maybe device_link_try_removing_from_wfs()?
I'll wait for us to agree on a better name here before I change this.
Quoted 11 lines
drivers/base/core.c struct device_link *device_link_add(struct device *consumer, · line 447
Quoted 8 lines
drivers/base/core.c +8
-0+{ +struct device *dev, *tmp; + +mutex_lock (&wfs_lock); +list_for_each_entry_safe (dev, tmp, &wait_for_suppliers, + links.needs_suppliers) +if (!dev->bus->add_links (dev)) +list_del_init (&dev->links.needs_suppliers);Empties
dev->links.needs_suppliers, but does not remove dev fromwait_for_supplierslist. Where does that happen?
I'll chalk this up to you having a long day or forgetting your coffee :) list_del_init() does both of those things because needs_suppliers is the node and wait_for_suppliers is the list.
Quoted 20 lines
Quoted 8 lines
drivers/base/core.c +3
-0+mutex_unlock (&wfs_lock); +} +static void
device_link_free(struct device_link *link){ while (refcount_dec_not_one(&link->rpm_active))drivers/base/core.c int device_links_check_suppliers(struct device *dev) · line 590
Quoted 9 lines
drivers/base/core.c +6
-0@@ -535,6 +582,19 @@ int device_links_check_suppliers(struct device *dev)struct device_link *link;int ret =0 ; +/* +* If a device is waiting for one or more suppliers (in +* wait_for_suppliers list), it is not ready to probe yet. So just +* return -EPROBE_DEFER without having to check the links with existing +* suppliers. +*/ Change comment to:
Quoted 4 lines
/*
* Device waiting for supplier to become available is not allowed
* to probe
*/
Po-tay-to. Po-tah-to? I think my comment is just as good.
Quoted 11 lines
drivers/base/core.c int device_links_check_suppliers(struct device *dev) · line 598
Quoted 8 lines
drivers/base/core.c +7
-0+mutex_lock (&wfs_lock); +if (!list_empty (&dev->links.needs_suppliers)) { +mutex_unlock (&wfs_lock); +return -EPROBE_DEFER ; + } +mutex_unlock (&wfs_lock); +device_links_write_lock ();Update
Documentation/driver-api/device_link.rstto reflect the check of&dev->links.needs_suppliersindevice_links_check_suppliers().
Thanks! Will do.
Quoted 41 lines
Quoted 30 lines
drivers/base/core.c +8
-0list_for_each_entry (link, &dev->links.suppliers, c_node) {@@ -812,6 +872,10 @@ static void device_links_purge(struct device *dev) {struct device_link *link, *ln; +mutex_lock (&wfs_lock); +list_del (&dev->links.needs_suppliers); +mutex_unlock (&wfs_lock); +/* * Delete all of the remaining links from this device to any other * devices (either consumers or suppliers). @@ -1673,6 +1737,7 @@ void device_initialize(struct device *dev)#endif INIT_LIST_HEAD (&dev->links.consumers);INIT_LIST_HEAD (&dev->links.suppliers); +INIT_LIST_HEAD (&dev->links.needs_suppliers); dev->links.status =DL_DEV_NO_DRIVER ; }EXPORT_SYMBOL_GPL (device_initialize);@@ -2108,6 +2173,24 @@ int device_add(struct device *dev) BUS_NOTIFY_ADD_DEVICE, dev);kobject_uevent (&dev->kobj,KOBJ_ADD ); + +/* +* Check if any of the other devices (consumers) have been waiting for drivers/base/core.c int device_add(struct device *dev) · line 2189
Quoted 10 lines
drivers/base/core.c +10
-0+* this device (supplier) to be added so that they can create a device +* link to it. +* +* This needs to happen after device_pm_add() because device_link_add() +* requires the supplier be registered before it's called. +* +* But this also needs to happe before bus_probe_device() to make sure +* waiting consumers can link to it before the driver is bound to the +* device and the driver sync_state callback is called for this device. +*/
Quoted 3 lines
/*
* Add links to dev from any dependent consumer that has dev on it's
* list of needed suppliers
There is no list of needed suppliers.
Quoted 6 lines
(links.needs_suppliers).
Device_pm_add()
* must have previously registered dev to allow the links to be added. *
* The consumer links must be created before dev is probed because the
*sync_statecallback for dev will use the consumer links. */
I think what I wrote is just as clear.
Quoted 18 lines
Quoted 17 lines
drivers/base/core.c +5
-0+device_link_check_waiting_consumers (); + +if (dev->bus && dev->bus->add_links && dev->bus->add_links (dev)) +device_link_wait_for_supplier (dev); +bus_probe_device (dev);if (parent)klist_add_tail(&dev->p->knode_parent,include/linux/device.h +0
-0@@ -78,6 +78,17 @@ extern void bus_remove_file(struct bus_type *, struct bus_attribute *);* -EPROBE_DEFER it will queue the device for deferred probing. * @uevent: Called when a device is added, removed, or a few other things * that generate uevents to add the environment variables.
Quoted 8 lines
include/linux/device.h extern void bus_remove_file(struct bus_type *, struct bus_attribute *); · line 86
Quoted 6 lines
include/linux/device.h +6
-0+* @add_links: Called, perhaps multiple times per device, after a device is +* added to this bus. The function is expected to create device +* links to all the suppliers of the input device that are +* available at the time this function is called. As in, the +* function should NOT stop at the first failed device link if +* other unlinked supplier devices are present in the system. * @add_links: Called after a device is added to this bus.
Why are you removing the "perhaps multiple times" part? that's true and that's how some of the other ops are documented.
Quoted 9 lines
The function is
* expected to create device links to all the suppliers of the * device that are available at the time this function is called. * The function must NOT stop at the first failed device link if * other unlinked supplier devices are present in the system. * If some suppliers are not yet available, this function will be * called again when the suppliers become available.but
add_links()not needed, so moving this comment toof_link_to_suppliers()
Sorry, I'm not sure I understand. Can you please explain what you are trying to say? of_link_to_suppliers() is just one implementation of add_links(). The comment above is try for any bus trying to implement add_links().
Quoted 25 lines
include/linux/device.h extern void bus_remove_file(struct bus_type *, struct bus_attribute *); · line 91
Quoted 7 lines
include/linux/device.h +5
-0+* +* Return 0 if device links have been successfully created to all +* the suppliers of this device. Return an error if some of the +* suppliers are not yet available and this function needs to be +* reattempted in the future. *
* Return 0 if device links have been successfully created to all * the suppliers of this device. Return an error if some of the * suppliers are not yet available.include/linux/device.h struct bus_type { · line 138
Quoted 10 lines
include/linux/device.h +1
-0* @probe: Called when a new device or driver add to this bus, and callback * the specific driver's probe to initial the matched device. * @remove: Called when a device removed from this bus. @@ -122,6 +133,7 @@ struct bus_type {int (*match)(struct device *dev,struct device_driver *drv);int (*uevent)(struct device *dev,struct kobj_uevent_env *env); +int (*add_links)(struct device *dev);
Quoted 8 lines
^^^^^^^^^ add_supplier ??? ^^^^^^^^^ add_suppliers ???^^^^^^^^^ link_suppliers ??? ^^^^^^^^^ add_supplier_dependency ??? ^^^^^^^^^ add_supplier_dependencies ???
add_links()not needed
add_links() was an intentional decision. There's no requirement that the bus should only create links from this device to its suppliers. If the bus also knows the consumers of this device (dev), then it can/should add those too. So, it shouldn't have "suppliers" in the name.
Quoted 12 lines
Quoted 11 lines
include/linux/device.h +0
-0int (*probe)(struct device *dev);int (*remove)(struct device *dev); void (*shutdown)(struct device *dev);include/linux/device.h +0
-0@@ -893,11 +905,13 @@ enum dl_dev_state {* struct dev_links_info - Device data related to device links. * @suppliers: List of links to supplier devices. * @consumers: List of links to consumer devices.
Quoted 4 lines
include/linux/device.h enum dl_dev_state { · line 908
Quoted 1 line
include/linux/device.h +1
-0+* @needs_suppliers: Hook to global list of devices waiting for suppliers. * @needs_suppliers: List of devices deferring probe until supplier drivers
* are successfully probed.
It's "need suppliers". As in, this is a device that "needs suppliers". So, no, this is not a list. This is a node in a list. And all "nodes in a list" are documented as "Hook" in rest of places in this file. So I think the documentation is correct as is.
Thanks for your review.
-Saravana
Re: [PATCH v7 1/7] driver core: Add support for linking devices during device addition
Quoted 31 lines · Saravana Kannan
Quoted 30 lines · Frank Rowand
Quoted 11 lines
Date: Tue, 23 Jul 2019 17:10:54 -0700
Subject: [PATCH v7 1/7] driver core: Add support for linking devices during
device addition
From: Saravana Kannan <[email protected]>When devices are added, the bus might want to create device links to track functional dependencies between supplier and consumer devices. This tracking of supplier-consumer relationship allows optimizing device probe order and tracking whether all consumers of a supplier are active. The
add_linksbus callback is added to support this.Change above to:
When devices are added, the bus may create device links to track which suppliers a consumer device depends upon. This
tracking of supplier-consumer relationship may be used to defer probing the driver of a consumer device before the driver(s) for its supplier device(s) are probed. It may also be used by a supplier driver to determine if all of its consumers have been successfully probed.
Theadd_linksbus callback is added to create the supplier device linksQuoted 5 lines
However, when consumer devices are added, they might not have a supplier device to link to despite needing mandatory resources/functionality from one or more suppliers. A
waiting_for_supplierslist is created to track such consumers and retry linking them when new devices get added.Change above to:
Quoted 39 lines
Quoted 5 lines
If a supplier device has not yet been created when the consumer device attempts to link it, the consumer device is added to the
wait_for_supplierslist. When supplier devices are created, the supplier device link will be added to the relevant consumer devices on thewait_for_supplierslist.I'll take these commit text suggestions if we decide to revert the entire series at the end of this review.
Quoted 29 lines
Quoted 21 lines
Signed-off-by Saravana Kannan <[email protected]>--- drivers/base/core.c | 83 ++++++++++++++++++++++++++++++++++++++++++ include/linux/device.h | 14 +++++++ 2 files changed, 97 insertions(+)drivers/base/core.c +2
-0@@ -44,6 +44,8 @@ early_param("sysfs.deprecated", sysfs_deprecated_setup);#endif /* Device links support. */ +static LIST_HEAD (wait_for_suppliers); +static DEFINE_MUTEX (wfs_lock);#ifdef CONFIG_SRCU static DEFINE_MUTEX (device_links_lock);drivers/base/core.c struct device_link *device_link_add(struct device *consumer, · line 408
Quoted 6 lines
drivers/base/core.c +2
-0@@ -401,6 +403,51 @@ struct device_link *device_link_add(struct device *consumer, }EXPORT_SYMBOL_GPL (device_link_add); +/** +* device_link_wait_for_supplier - Mark device as waiting for supplier *
device_link_wait_for_supplier- Add device towait_for_supplierslist
As a meta-comment, I found this series very hard to understand in the context of reading the new code for the first time. When I read the code again in six months or a year or two years it will not be in near term memory and it will be as if I am reading it for the first time. A lot of my suggestions for changes of names are in that context -- the current names may be fine when one has recently read the code, but not so much when trying to read the whole thing again with a blank mind.
The code also inherits a good deal of complexity because it does not stand alone in a nice discrete chunk, but instead delicately weaves into a more complex body of code.
When I was trying to understand the code, I wrote a lot of additional comments within my reply email to provide myself context, information about various things, and questions that I needed to answer (or if I could not answer to then ask you). Then I ended up being able to remove many of those notes before sending the reply.
Quoted 6 lines
I intentionally chose "Mark device..." because that's a better description of the semantics of the function instead of trying to describe the implementation. Whether I'm using a linked list or some other data structure should not be the one line documentation of a function. Unless the function is explicitly about operating on that specific data structure.
I agree with the intent of trying to describe the semantics of a function, especially at the API level where other systems (or drivers) would be using the function. But for this case the function is at the implementation level and describing explicitly what it is doing makes this much more readable for me.
I also find "Mark device" to be vague and not descriptive of what the intent is.
Quoted 47 lines
Quoted 36 lines
Quoted 25 lines
drivers/base/core.c +20
-0+* @consumer: Consumer device +* +* Marks the consumer device as waiting for suppliers to become available. The +* consumer device will never be probed until it's unmarked as waiting for +* suppliers. The caller is responsible for adding the link to the supplier +* once the supplier device is present. +* +* This function is NOT meant to be called from the probe function of the +* consumer but rather from code that creates/adds the consumer device. +*/ +static void device_link_wait_for_supplier (struct device *consumer) +{ +mutex_lock (&wfs_lock); +list_add_tail (&consumer->links.needs_suppliers, &wait_for_suppliers); +mutex_unlock (&wfs_lock); +} + +/** +* device_link_check_waiting_consumers - Try to remove from supplier wait list +* drivers/base/core.c struct device_link *device_link_add(struct device *consumer, · line 437
Quoted 10 lines
drivers/base/core.c +10
-0+* Loops through all consumers waiting on suppliers and tries to add all their +* supplier links. If that succeeds, the consumer device is unmarked as waiting +* for suppliers. Otherwise, they are left marked as waiting on suppliers, +* +* The add_links bus callback is expected to return 0 if it has found and added +* all the supplier links for the consumer device. It should return an error if +* it isn't able to do so. +* +* The caller of device_link_wait_for_supplier() is expected to call this once +* it's aware of potential suppliers becoming available. Quoted 6 lines
Change above comment to:
*
device_link_add_supplier_links- add links from consumer devices to
* supplier devices, leaving any consumer
* with inactive suppliers on the
*wait_for_supplierslistI didn't know that the first one line comment could span multiple lines. Good to know.
Quoted 6 lines
Quoted 1 line
* Scan all consumer devices in the devicetree.
This function doesn't have anything to do with devicetree. I've intentionally kept all OF related parts out of the driver/core because I hope that other busses can start using this feature too. So I can't take this bit.
My comment is left over from when I was taking notes, trying to understand the code.
At the moment, only devicetree is used as a source of the dependency information. The comment would better be re-phrased as:
* Scan all consumer devices in the firmware description of the hardware topology
I did not ask why this feature is tied to _only_ the platform bus, but will now.
I do not know of any reason that a consumer / supplier relationship can not be between devices on different bus types. Do you know of such a reason?
Quoted 11 lines
Quoted 9 lines
For any supplier device that
* is not already linked to the consumer device, add the supplier to the
* consumer device's device links.
*
* If all of a consumer device's suppliers are available then the consumer
* is removed from thewait_for_supplierslist (if previously on the list).
* Otherwise the consumer is added to thewait_for_supplierslist (if not
* already on the list).Honestly, I don't think this is any better than what I already have.
Note that my version of these comments was written while I was reading the code, and did not have any big picture understanding yet. This will likely also be the mind set of most everyone who reads this code in the future, once it is woven into the kernel.
If you don't like the change, I can revisit it in a later version of the patch set.
Quoted 15 lines
Quoted 15 lines
* The
add_linksbus callback must return 0 if it has found and added all
* the supplier links for the consumer device. It must return an error if
* it is not able to do so.
*
* The caller ofdevice_link_wait_for_supplier()is expected to call this once
* it is aware of potential suppliers becoming available.drivers/base/core.c struct device_link *device_link_add(struct device *consumer, · line 442
Quoted 2 lines
drivers/base/core.c +2
-0+*/ +static void device_link_check_waiting_consumers (void )Function name is misleading and hides side effects.
Quoted 7 lines
Quoted 4 lines
I have not come up with a name that does not hide side effects, but a better name would be:
device_link_add_supplier_links()I kinda agree that it could afford a better name. The current name is too similar to
device_links_check_suppliers()and I never liked that.
Naming new fields or variables related to device links looks pretty challenging to me, because of the desire to be part of device links and not a wart pasted on the side. So I share the pain in trying to find good names.
Quoted 2 lines
Maybe
device_link_add_missing_suppliers()?
My first reaction was "yes, that sounds good". But then I stopped and tried to read the name out of context. The name is not adding the missing suppliers, it is saving the information that a supplier is not yet available (eg, is "missing"). I struggled in coming up with the name that I suggested. We can keep thinking.
Quoted 2 lines
I don't think we need "links" repeated twice in the function name.
Yeah, I didn't like that either.
Quoted 3 lines
With this suggestion, what side effect is hidden in your opinion? That the fully linked consumer is removed from the "waiting for suppliers" list?
The side effect is that the function does not merely do a check. It also adds missing suppliers to a list.
Quoted 2 lines
Maybe
device_link_try_removing_from_wfs()?
I like that, other than the fact that it still does not provide a clue that the function is potentially adding suppliers to a list. I think part of the challenge is that the function does two things: (1) a check, and (2) potentially adding missing suppliers to a list. Maybe a simple one line comment at the call site, something like:
/* adds missing suppliers to wfs */
Quoted 15 lines
I'll wait for us to agree on a better name here before I change this.
Quoted 11 lines
drivers/base/core.c struct device_link *device_link_add(struct device *consumer, · line 447
Quoted 8 lines
drivers/base/core.c +8
-0+{ +struct device *dev, *tmp; + +mutex_lock (&wfs_lock); +list_for_each_entry_safe (dev, tmp, &wait_for_suppliers, + links.needs_suppliers) +if (!dev->bus->add_links (dev)) +list_del_init (&dev->links.needs_suppliers);Empties
dev->links.needs_suppliers, but does not remove dev fromwait_for_supplierslist. Where does that happen?
Quoted 3 lines
I'll chalk this up to you having a long day or forgetting your coffee :)
list_del_init()does both of those things becauseneeds_suppliersis the node andwait_for_suppliersis the list.
Yes, brain mis-fire on my part. I'll have to go back and look at the list related code again.
Quoted 21 lines
Quoted 21 lines
Quoted 9 lines
drivers/base/core.c +3
-0+mutex_unlock (&wfs_lock); +} +static void
device_link_free(struct device_link *link){ while (refcount_dec_not_one(&link->rpm_active))drivers/base/core.c int device_links_check_suppliers(struct device *dev) · line 590
Quoted 9 lines
drivers/base/core.c +6
-0@@ -535,6 +582,19 @@ int device_links_check_suppliers(struct device *dev)struct device_link *link;int ret =0 ; +/* +* If a device is waiting for one or more suppliers (in +* wait_for_suppliers list), it is not ready to probe yet. So just +* return -EPROBE_DEFER without having to check the links with existing +* suppliers. +*/ Change comment to:
Quoted 6 lines
Quoted 4 lines
/*
* Device waiting for supplier to become available is not allowed
* to probe
*/Po-tay-to. Po-tah-to? I think my comment is just as good.
If just as good and shorter, then better.
Also the original says "it is not ready to probe". That is not correct. It is ready to probe, it is just that the probe attempt will return -EPROBE_DEFER. Nit picky on my part, but tiny things like that mean I have to think harder. I have to think "why is it not ready to probe?". Maybe my version should have instead been something like:
* Device waiting for supplier to become available will return
* -EPROBE_DEFER if probed. Avoid the unneeded processing.
Quoted 56 lines
Quoted 10 lines
drivers/base/core.c int device_links_check_suppliers(struct device *dev) · line 598
Quoted 9 lines
drivers/base/core.c +7
-0+mutex_lock (&wfs_lock); +if (!list_empty (&dev->links.needs_suppliers)) { +mutex_unlock (&wfs_lock); +return -EPROBE_DEFER ; + } +mutex_unlock (&wfs_lock); +device_links_write_lock ();Quoted 2 lines
Update
Documentation/driver-api/device_link.rstto reflect the check of&dev->links.needs_suppliersindevice_links_check_suppliers().Thanks! Will do.
Quoted 41 lines
Quoted 40 lines
drivers/base/core.c +18
-0list_for_each_entry (link, &dev->links.suppliers, c_node) {@@ -812,6 +872,10 @@ static void device_links_purge(struct device *dev) {struct device_link *link, *ln; +mutex_lock (&wfs_lock); +list_del (&dev->links.needs_suppliers); +mutex_unlock (&wfs_lock); +/* * Delete all of the remaining links from this device to any other * devices (either consumers or suppliers). @@ -1673,6 +1737,7 @@ void device_initialize(struct device *dev)#endif INIT_LIST_HEAD (&dev->links.consumers);INIT_LIST_HEAD (&dev->links.suppliers); +INIT_LIST_HEAD (&dev->links.needs_suppliers); dev->links.status =DL_DEV_NO_DRIVER ; }EXPORT_SYMBOL_GPL (device_initialize);@@ -2108,6 +2173,24 @@ int device_add(struct device *dev) BUS_NOTIFY_ADD_DEVICE, dev);kobject_uevent (&dev->kobj,KOBJ_ADD ); + +/* +* Check if any of the other devices (consumers) have been waiting for +* this device (supplier) to be added so that they can create a device +* link to it. +* +* This needs to happen after device_pm_add() because device_link_add() +* requires the supplier be registered before it's called. +* +* But this also needs to happe before bus_probe_device() to make sure +* waiting consumers can link to it before the driver is bound to the +* device and the driver sync_state callback is called for this device. +*/
Quoted 5 lines
Quoted 3 lines
/*
* Add links to dev from any dependent consumer that has dev on it's
* list of needed suppliersThere is no list of needed suppliers.
"the other devices (consumers) have been waiting for this device (supplier)". Isn't that a list of needed suppliers?
Quoted 9 lines
Quoted 7 lines
(links.needs_suppliers).
Device_pm_add()
* must have previously registered dev to allow the links to be added. *
* The consumer links must be created before dev is probed because the
*sync_statecallback for dev will use the consumer links. */I think what I wrote is just as clear.
The original comment is vague. It does not explain why consumer links must be created before the probe. I had to go off and read other code to determine why that is true.
And again, brevity is better if otherwise just as clear.
Quoted 26 lines
Quoted 26 lines
Quoted 25 lines
drivers/base/core.c +5
-0+device_link_check_waiting_consumers (); + +if (dev->bus && dev->bus->add_links && dev->bus->add_links (dev)) +device_link_wait_for_supplier (dev); +bus_probe_device (dev);if (parent)klist_add_tail(&dev->p->knode_parent,include/linux/device.h +6
-0@@ -78,6 +78,17 @@ extern void bus_remove_file(struct bus_type *, struct bus_attribute *);* -EPROBE_DEFER it will queue the device for deferred probing. * @uevent: Called when a device is added, removed, or a few other things * that generate uevents to add the environment variables. +* @add_links: Called, perhaps multiple times per device, after a device is +* added to this bus. The function is expected to create device +* links to all the suppliers of the input device that are +* available at the time this function is called. As in, the +* function should NOT stop at the first failed device link if +* other unlinked supplier devices are present in the system.
Quoted 4 lines
Quoted 1 line
* @add_links: Called after a device is added to this bus.
Why are you removing the "perhaps multiple times" part? that's true and that's how some of the other ops are documented.
I didn't remove it. I rephrased it with a little bit more explanation as "If some suppliers are not yet available, this function will be called again when the suppliers become available." (below).
Quoted 9 lines
Quoted 9 lines
The function is
* expected to create device links to all the suppliers of the * device that are available at the time this function is called. * The function must NOT stop at the first failed device link if * other unlinked supplier devices are present in the system. * If some suppliers are not yet available, this function will be * called again when the suppliers become available.
Quoted 6 lines
Quoted 1 line
but
add_links()not needed, so moving this comment toof_link_to_suppliers()Sorry, I'm not sure I understand. Can you please explain what you are trying to say?
of_link_to_suppliers()is just one implementation ofadd_links(). The comment above is try for any bus trying to implementadd_links().
This is conflating bus with the source of the firmware description of the hardware topology. For drivers that use various APIs to access firmware description of topology that may be either devicetree or ACPI the access is done via fwnode_operations, based on struct device.fwnode (if I recall properly).
I failed to completely address why add_links() is not needed. The answer is that there should be a single function called for all buses. Then the proper firmware data source would be accessed via a struct fwnode_operations.
I think I left this out because I had not yet asked why this feature is tied only to the platform bus. Which I asked earlier in this reply.
Quoted 31 lines
Quoted 31 lines
include/linux/device.h extern void bus_remove_file(struct bus_type *, struct bus_attribute *); · line 91
Quoted 8 lines
include/linux/device.h +5
-0+* +* Return 0 if device links have been successfully created to all +* the suppliers of this device. Return an error if some of the +* suppliers are not yet available and this function needs to be +* reattempted in the future. *
* Return 0 if device links have been successfully created to all * the suppliers of this device. Return an error if some of the * suppliers are not yet available.include/linux/device.h struct bus_type { · line 138
Quoted 10 lines
include/linux/device.h +0
-0* @probe: Called when a new device or driver add to this bus, and callback * the specific driver's probe to initial the matched device. * @remove: Called when a device removed from this bus.
include/linux/device.h +1
-0@@ -122,6 +133,7 @@ struct bus_type {int (*match)(struct device *dev,struct device_driver *drv);int (*uevent)(struct device *dev,struct kobj_uevent_env *env); +int (*add_links)(struct device *dev);^^^^^^^^^ add_supplier ??? ^^^^^^^^^ add_suppliers ???^^^^^^^^^ link_suppliers ???
Quoted 8 lines
Quoted 3 lines
^^^^^^^^^ add_supplier_dependency ??? ^^^^^^^^^ add_supplier_dependencies ???
add_links()not needed
add_links()was an intentional decision. There's no requirement that the bus should only create links from this device to its suppliers. If the bus also knows the consumers of this device (dev), then it can/should add those too.
Is creating links to consumers of this device implemented in this patch series? If so, I overlooked it and will have to consider how that fits in to the design.
Quoted 17 lines
So, it shouldn't have "suppliers" in the
name.Quoted 14 lines
Quoted 13 lines
include/linux/device.h +0
-0int (*probe)(struct device *dev);int (*remove)(struct device *dev); void (*shutdown)(struct device *dev);include/linux/device.h +1
-0@@ -893,11 +905,13 @@ enum dl_dev_state {* struct dev_links_info - Device data related to device links. * @suppliers: List of links to supplier devices. * @consumers: List of links to consumer devices. +* @needs_suppliers: Hook to global list of devices waiting for suppliers.
Quoted 7 lines
Quoted 2 lines
* @needs_suppliers: List of devices deferring probe until supplier drivers
* are successfully probed.It's "need suppliers". As in, this is a device that "needs suppliers". So, no, this is not a list. This is a node in a list. And all "nodes in a list" are documented as "Hook" in rest of places in this file. So I think the documentation is correct as is.
Aha, I got confused about that while trying to keep everything straight.
Somehow I managed to conflate needs_suppliers with the links between consumers and suppliers that are create via device_link_add().
So original comment is fine.
It is getting late, so I'll continue with patches 2 and 3 tomorrow.
-Frank
Quoted 7 lines
Thanks for your review.
-Saravana
Re: [PATCH v7 1/7] driver core: Add support for linking devices during device addition
Quoted 73 lines · Frank Rowand
Quoted 31 lines · Saravana Kannan
Quoted 30 lines · Frank Rowand
Quoted 11 lines
Date: Tue, 23 Jul 2019 17:10:54 -0700
Subject: [PATCH v7 1/7] driver core: Add support for linking devices during
device addition
From: Saravana Kannan <[email protected]>When devices are added, the bus might want to create device links to track functional dependencies between supplier and consumer devices. This tracking of supplier-consumer relationship allows optimizing device probe order and tracking whether all consumers of a supplier are active. The
add_linksbus callback is added to support this.Change above to:
When devices are added, the bus may create device links to track which suppliers a consumer device depends upon. This
tracking of supplier-consumer relationship may be used to defer probing the driver of a consumer device before the driver(s) for its supplier device(s) are probed. It may also be used by a supplier driver to determine if all of its consumers have been successfully probed.
Theadd_linksbus callback is added to create the supplier device linksQuoted 5 lines
However, when consumer devices are added, they might not have a supplier device to link to despite needing mandatory resources/functionality from one or more suppliers. A
waiting_for_supplierslist is created to track such consumers and retry linking them when new devices get added.Change above to:
Quoted 39 lines
Quoted 5 lines
If a supplier device has not yet been created when the consumer device attempts to link it, the consumer device is added to the
wait_for_supplierslist. When supplier devices are created, the supplier device link will be added to the relevant consumer devices on thewait_for_supplierslist.I'll take these commit text suggestions if we decide to revert the entire series at the end of this review.
Quoted 29 lines
Quoted 21 lines
Signed-off-by Saravana Kannan <[email protected]>--- drivers/base/core.c | 83 ++++++++++++++++++++++++++++++++++++++++++ include/linux/device.h | 14 +++++++ 2 files changed, 97 insertions(+)drivers/base/core.c +2
-0@@ -44,6 +44,8 @@ early_param("sysfs.deprecated", sysfs_deprecated_setup);#endif /* Device links support. */ +static LIST_HEAD (wait_for_suppliers); +static DEFINE_MUTEX (wfs_lock);#ifdef CONFIG_SRCU static DEFINE_MUTEX (device_links_lock);drivers/base/core.c struct device_link *device_link_add(struct device *consumer, · line 408
Quoted 6 lines
drivers/base/core.c +2
-0@@ -401,6 +403,51 @@ struct device_link *device_link_add(struct device *consumer, }EXPORT_SYMBOL_GPL (device_link_add); +/** +* device_link_wait_for_supplier - Mark device as waiting for supplier *
device_link_wait_for_supplier- Add device towait_for_supplierslist
Quoted 7 lines
As a meta-comment, I found this series very hard to understand in the context of reading the new code for the first time. When I read the code again in six months or a year or two years it will not be in near term memory and it will be as if I am reading it for the first time. A lot of my suggestions for changes of names are in that context -- the current names may be fine when one has recently read the code, but not so much when trying to read the whole thing again with a blank mind.
Thanks for the context.
Quoted 3 lines
The code also inherits a good deal of complexity because it does not stand alone in a nice discrete chunk, but instead delicately weaves into a more complex body of code.
I'll take this as a compliment :)
Quoted 14 lines
When I was trying to understand the code, I wrote a lot of additional comments within my reply email to provide myself context, information about various things, and questions that I needed to answer (or if I could not answer to then ask you). Then I ended up being able to remove many of those notes before sending the reply.
Quoted 6 lines
I intentionally chose "Mark device..." because that's a better description of the semantics of the function instead of trying to describe the implementation. Whether I'm using a linked list or some other data structure should not be the one line documentation of a function. Unless the function is explicitly about operating on that specific data structure.
Quoted 5 lines
I agree with the intent of trying to describe the semantics of a function, especially at the API level where other systems (or drivers) would be using the function. But for this case the function is at the implementation level and describing explicitly what it is doing makes this much more readable for me.
Are you distinguishing between API level vs implementation level based on the function being "static"/not exported? I believe the earlier version of this series had this function as an exported API. So maybe that's why I had it as "Mark device".
Quoted 57 lines
I also find "Mark device" to be vague and not descriptive of what the intent is.
Quoted 47 lines
Quoted 36 lines
Quoted 25 lines
drivers/base/core.c +20
-0+* @consumer: Consumer device +* +* Marks the consumer device as waiting for suppliers to become available. The +* consumer device will never be probed until it's unmarked as waiting for +* suppliers. The caller is responsible for adding the link to the supplier +* once the supplier device is present. +* +* This function is NOT meant to be called from the probe function of the +* consumer but rather from code that creates/adds the consumer device. +*/ +static void device_link_wait_for_supplier (struct device *consumer) +{ +mutex_lock (&wfs_lock); +list_add_tail (&consumer->links.needs_suppliers, &wait_for_suppliers); +mutex_unlock (&wfs_lock); +} + +/** +* device_link_check_waiting_consumers - Try to remove from supplier wait list +* drivers/base/core.c struct device_link *device_link_add(struct device *consumer, · line 437
Quoted 10 lines
drivers/base/core.c +10
-0+* Loops through all consumers waiting on suppliers and tries to add all their +* supplier links. If that succeeds, the consumer device is unmarked as waiting +* for suppliers. Otherwise, they are left marked as waiting on suppliers, +* +* The add_links bus callback is expected to return 0 if it has found and added +* all the supplier links for the consumer device. It should return an error if +* it isn't able to do so. +* +* The caller of device_link_wait_for_supplier() is expected to call this once +* it's aware of potential suppliers becoming available. Quoted 6 lines
Change above comment to:
*
device_link_add_supplier_links- add links from consumer devices to
* supplier devices, leaving any consumer
* with inactive suppliers on the
*wait_for_supplierslistI didn't know that the first one line comment could span multiple lines. Good to know.
Quoted 6 lines
Quoted 1 line
* Scan all consumer devices in the devicetree.
This function doesn't have anything to do with devicetree. I've intentionally kept all OF related parts out of the driver/core because I hope that other busses can start using this feature too. So I can't take this bit.
Quoted 8 lines
My comment is left over from when I was taking notes, trying to understand the code.
At the moment, only devicetree is used as a source of the dependency information. The comment would better be re-phrased as:
* Scan all consumer devices in the firmware description of the hardware topology
Ok
Quoted 1 line
I did not ask why this feature is tied to _only_ the platform bus, but will now.
Because devicetree and platform bus the only ones I'm familiar with. If other busses want to add this, I'd be happy to help with code and/or direction/review. But I won't pretend to know anything about ACPI.
Quoted 2 lines
I do not know of any reason that a consumer / supplier relationship can not be between devices on different bus types. Do you know of such a reason?
Yes, it's hypothetically possible. But I haven't seen such a relationship being defined in DT. Nor somewhere else where this might be captured. So, how common/realistic is it?
Quoted 12 lines
Quoted 11 lines
Quoted 9 lines
For any supplier device that
* is not already linked to the consumer device, add the supplier to the
* consumer device's device links.
*
* If all of a consumer device's suppliers are available then the consumer
* is removed from thewait_for_supplierslist (if previously on the list).
* Otherwise the consumer is added to thewait_for_supplierslist (if not
* already on the list).Honestly, I don't think this is any better than what I already have.
Quoted 7 lines
Note that my version of these comments was written while I was reading the code, and did not have any big picture understanding yet. This will likely also be the mind set of most everyone who reads this code in the future, once it is woven into the kernel.
If you don't like the change, I can revisit it in a later version of the patch set.
I'll take in all the ones I feel are reasonable or don't feel strongly about. We can revisit the rest later.
Quoted 29 lines
Quoted 15 lines
Quoted 15 lines
* The
add_linksbus callback must return 0 if it has found and added all
* the supplier links for the consumer device. It must return an error if
* it is not able to do so.
*
* The caller ofdevice_link_wait_for_supplier()is expected to call this once
* it is aware of potential suppliers becoming available.drivers/base/core.c struct device_link *device_link_add(struct device *consumer, · line 442
Quoted 2 lines
drivers/base/core.c +2
-0+*/ +static void device_link_check_waiting_consumers (void )Function name is misleading and hides side effects.
Quoted 7 lines
Quoted 4 lines
I have not come up with a name that does not hide side effects, but a better name would be:
device_link_add_supplier_links()I kinda agree that it could afford a better name. The current name is too similar to
device_links_check_suppliers()and I never liked that.Naming new fields or variables related to device links looks pretty challenging to me, because of the desire to be part of device links and not a wart pasted on the side. So I share the pain in trying to find good names.
Quoted 1 line
Quoted 7 lines
Quoted 1 line
Maybe
device_link_add_missing_suppliers()?My first reaction was "yes, that sounds good". But then I stopped and tried to read the name out of context. The name is not adding the missing suppliers, it is saving the information that a supplier is not yet available (eg, is "missing"). I struggled in coming up with the name that I suggested. We can keep thinking.
No, this function _IS_ about adding links to suppliers. These consumers were "saved" as "not yet having the supplier" earlier by device_link_wait_for_supplier(). This function doesn't do that. This function is just trying to see if those missing suppliers are present now and if so adding a link to them from the "saved" consumers. I think device_link_add_missing_suppliers() is actually a pretty good name. Let me know what you think now.
Quoted 8 lines
Quoted 4 lines
I don't think we need "links" repeated twice in the function name.
Yeah, I didn't like that either.
Quoted 6 lines
Quoted 3 lines
With this suggestion, what side effect is hidden in your opinion? That the fully linked consumer is removed from the "waiting for suppliers" list?
The side effect is that the function does not merely do a check. It also adds missing suppliers to a list.
No, it doesn't do that. I can't keep a list of things that aren't allocated yet :). In the whole patch series, we only keep a list of things (consumers) that are waiting on other things (missing suppliers).
Quoted 5 lines
Quoted 2 lines
Maybe
device_link_try_removing_from_wfs()?I like that, other than the fact that it still does not provide a clue that the function is potentially adding suppliers to a list.
It doesn't. How would you add a supplier device to a list if the device itself isn't there? :)
Quoted 126 lines
I think
part of the challenge is that the function does two things: (1) a check, and (2) potentially adding missing suppliers to a list. Maybe a simple one line comment at the call site, something like:/* adds missing suppliers to wfs */Quoted 15 lines
I'll wait for us to agree on a better name here before I change this.
Quoted 11 lines
drivers/base/core.c struct device_link *device_link_add(struct device *consumer, · line 447
Quoted 8 lines
drivers/base/core.c +8
-0+{ +struct device *dev, *tmp; + +mutex_lock (&wfs_lock); +list_for_each_entry_safe (dev, tmp, &wait_for_suppliers, + links.needs_suppliers) +if (!dev->bus->add_links (dev)) +list_del_init (&dev->links.needs_suppliers);Empties
dev->links.needs_suppliers, but does not remove dev fromwait_for_supplierslist. Where does that happen?Quoted 3 lines
I'll chalk this up to you having a long day or forgetting your coffee :)
list_del_init()does both of those things becauseneeds_suppliersis the node andwait_for_suppliersis the list.Yes, brain mis-fire on my part. I'll have to go back and look at the list related code again.
Quoted 21 lines
Quoted 21 lines
Quoted 9 lines
drivers/base/core.c +3
-0+mutex_unlock (&wfs_lock); +} +static void
device_link_free(struct device_link *link){ while (refcount_dec_not_one(&link->rpm_active))drivers/base/core.c int device_links_check_suppliers(struct device *dev) · line 590
Quoted 9 lines
drivers/base/core.c +6
-0@@ -535,6 +582,19 @@ int device_links_check_suppliers(struct device *dev)struct device_link *link;int ret =0 ; +/* +* If a device is waiting for one or more suppliers (in +* wait_for_suppliers list), it is not ready to probe yet. So just +* return -EPROBE_DEFER without having to check the links with existing +* suppliers. +*/ Change comment to:
Quoted 6 lines
Quoted 4 lines
/*
* Device waiting for supplier to become available is not allowed
* to probe
*/Po-tay-to. Po-tah-to? I think my comment is just as good.
If just as good and shorter, then better.
Also the original says "it is not ready to probe". That is not correct. It is ready to probe, it is just that the probe attempt will return -EPROBE_DEFER. Nit picky on my part, but tiny things like that mean I have to think harder. I have to think "why is it not ready to probe?". Maybe my version should have instead been something like:
* Device waiting for supplier to become available will return
* -EPROBE_DEFER if probed. Avoid the unneeded processing.Quoted 56 lines
Quoted 10 lines
drivers/base/core.c int device_links_check_suppliers(struct device *dev) · line 598
Quoted 9 lines
drivers/base/core.c +7
-0+mutex_lock (&wfs_lock); +if (!list_empty (&dev->links.needs_suppliers)) { +mutex_unlock (&wfs_lock); +return -EPROBE_DEFER ; + } +mutex_unlock (&wfs_lock); +device_links_write_lock ();Quoted 2 lines
Update
Documentation/driver-api/device_link.rstto reflect the check of&dev->links.needs_suppliersindevice_links_check_suppliers().Thanks! Will do.
Quoted 41 lines
Quoted 40 lines
drivers/base/core.c +18
-0list_for_each_entry (link, &dev->links.suppliers, c_node) {@@ -812,6 +872,10 @@ static void device_links_purge(struct device *dev) {struct device_link *link, *ln; +mutex_lock (&wfs_lock); +list_del (&dev->links.needs_suppliers); +mutex_unlock (&wfs_lock); +/* * Delete all of the remaining links from this device to any other * devices (either consumers or suppliers). @@ -1673,6 +1737,7 @@ void device_initialize(struct device *dev)#endif INIT_LIST_HEAD (&dev->links.consumers);INIT_LIST_HEAD (&dev->links.suppliers); +INIT_LIST_HEAD (&dev->links.needs_suppliers); dev->links.status =DL_DEV_NO_DRIVER ; }EXPORT_SYMBOL_GPL (device_initialize);@@ -2108,6 +2173,24 @@ int device_add(struct device *dev) BUS_NOTIFY_ADD_DEVICE, dev);kobject_uevent (&dev->kobj,KOBJ_ADD ); + +/* +* Check if any of the other devices (consumers) have been waiting for +* this device (supplier) to be added so that they can create a device +* link to it. +* +* This needs to happen after device_pm_add() because device_link_add() +* requires the supplier be registered before it's called. +* +* But this also needs to happe before bus_probe_device() to make sure +* waiting consumers can link to it before the driver is bound to the +* device and the driver sync_state callback is called for this device. +*/
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
/*
* Add links to dev from any dependent consumer that has dev on it's
* list of needed suppliersThere is no list of needed suppliers.
"the other devices (consumers) have been waiting for this device (supplier)". Isn't that a list of needed suppliers?
No, that's a list of consumers that needs_suppliers.
Quoted 75 lines
Quoted 9 lines
Quoted 7 lines
(links.needs_suppliers).
Device_pm_add()
* must have previously registered dev to allow the links to be added. *
* The consumer links must be created before dev is probed because the
*sync_statecallback for dev will use the consumer links. */I think what I wrote is just as clear.
The original comment is vague. It does not explain why consumer links must be created before the probe. I had to go off and read other code to determine why that is true.
And again, brevity is better if otherwise just as clear.
Quoted 26 lines
Quoted 26 lines
Quoted 25 lines
drivers/base/core.c +5
-0+device_link_check_waiting_consumers (); + +if (dev->bus && dev->bus->add_links && dev->bus->add_links (dev)) +device_link_wait_for_supplier (dev); +bus_probe_device (dev);if (parent)klist_add_tail(&dev->p->knode_parent,include/linux/device.h +6
-0@@ -78,6 +78,17 @@ extern void bus_remove_file(struct bus_type *, struct bus_attribute *);* -EPROBE_DEFER it will queue the device for deferred probing. * @uevent: Called when a device is added, removed, or a few other things * that generate uevents to add the environment variables. +* @add_links: Called, perhaps multiple times per device, after a device is +* added to this bus. The function is expected to create device +* links to all the suppliers of the input device that are +* available at the time this function is called. As in, the +* function should NOT stop at the first failed device link if +* other unlinked supplier devices are present in the system. Quoted 4 lines
Quoted 1 line
* @add_links: Called after a device is added to this bus.
Why are you removing the "perhaps multiple times" part? that's true and that's how some of the other ops are documented.
I didn't remove it. I rephrased it with a little bit more explanation as "If some suppliers are not yet available, this function will be called again when the suppliers become available." (below).
Quoted 9 lines
Quoted 9 lines
The function is
* expected to create device links to all the suppliers of the * device that are available at the time this function is called. * The function must NOT stop at the first failed device link if * other unlinked supplier devices are present in the system. * If some suppliers are not yet available, this function will be * called again when the suppliers become available.Quoted 6 lines
Quoted 1 line
but
add_links()not needed, so moving this comment toof_link_to_suppliers()Sorry, I'm not sure I understand. Can you please explain what you are trying to say?
of_link_to_suppliers()is just one implementation ofadd_links(). The comment above is try for any bus trying to implementadd_links().This is conflating bus with the source of the firmware description of the hardware topology. For drivers that use various APIs to access firmware description of topology that may be either devicetree or ACPI the access is done via
fwnode_operations, based on struct device.fwnode (if I recall properly).
Quoted 6 lines
I failed to completely address why
add_links()is not needed. The answer is that there should be a single function called for all buses. Then the proper firmware data source would be accessed via a structfwnode_operations.I think I left this out because I had not yet asked why this feature is tied only to the platform bus. Which I asked earlier in this reply.
Thanks for the pointer about fwnode and fwnode_operations. I wasn't aware of those. I see where you are going with this. I see a couple of problems with this approach though:
1. How you interpret the properties of a fwnode is specific to the fw type. The clocks DT property isn't going to have the same definition in ACPI or some other firmware. Heck, I don't know if ACPI even has a clocks like property. So have one function to parse all the FW types doesn't make a lot of sense.
2. If this common code is implemented as part of driver/base/, then at a minimum, I'll have to check if a fwnode is a DT node before I start interpreting the properties of a device's fwnode. But that means I'll have to include linux/of.h to use is_of_node(). I don't like having driver/base code depend on OF or platform or ACPI headers.
3. The supplier info doesn't always need to come from a firmware. So I don't want to limit it to that?
Also, I don't necessarily see this as conflating firmware (DT, ACPI, etc) with the bus (platform bus, ACPI bus, PCI bus). Whoever creates the device seems like the entity best suited to figure out the suppliers of the device (apart from the driver, obviously). So the bus deciding the suppliers doesn't seem wrong to me.
In this specific case, I'm trying to address DT for now and leaving ACPI to whoever else wants to add device links based on ACPI data. Most OF/DT based devices end up in platform bus. So I'm just handling this in platform bus. If some other person wants this to work for ACPI bus or PCI bus, they are welcome to implement add_links() for those busses? I'm nowhere close to an expert on those.
Quoted 70 lines
Quoted 31 lines
Quoted 31 lines
include/linux/device.h extern void bus_remove_file(struct bus_type *, struct bus_attribute *); · line 91
Quoted 8 lines
include/linux/device.h +5
-0+* +* Return 0 if device links have been successfully created to all +* the suppliers of this device. Return an error if some of the +* suppliers are not yet available and this function needs to be +* reattempted in the future. *
* Return 0 if device links have been successfully created to all * the suppliers of this device. Return an error if some of the * suppliers are not yet available.include/linux/device.h struct bus_type { · line 138
Quoted 10 lines
include/linux/device.h +0
-0* @probe: Called when a new device or driver add to this bus, and callback * the specific driver's probe to initial the matched device. * @remove: Called when a device removed from this bus.
include/linux/device.h +1
-0@@ -122,6 +133,7 @@ struct bus_type {int (*match)(struct device *dev,struct device_driver *drv);int (*uevent)(struct device *dev,struct kobj_uevent_env *env); +int (*add_links)(struct device *dev);^^^^^^^^^ add_supplier ??? ^^^^^^^^^ add_suppliers ???^^^^^^^^^ link_suppliers ???Quoted 8 lines
Quoted 3 lines
^^^^^^^^^ add_supplier_dependency ??? ^^^^^^^^^ add_supplier_dependencies ???
add_links()not needed
add_links()was an intentional decision. There's no requirement that the bus should only create links from this device to its suppliers. If the bus also knows the consumers of this device (dev), then it can/should add those too.Is creating links to consumers of this device implemented in this patch series? If so, I overlooked it and will have to consider how that fits in to the design.
Quoted 17 lines
So, it shouldn't have "suppliers" in the
name.Quoted 14 lines
Quoted 13 lines
include/linux/device.h +0
-0int (*probe)(struct device *dev);int (*remove)(struct device *dev); void (*shutdown)(struct device *dev);include/linux/device.h +1
-0@@ -893,11 +905,13 @@ enum dl_dev_state {* struct dev_links_info - Device data related to device links. * @suppliers: List of links to supplier devices. * @consumers: List of links to consumer devices. +* @needs_suppliers: Hook to global list of devices waiting for suppliers. Quoted 7 lines
Quoted 2 lines
* @needs_suppliers: List of devices deferring probe until supplier drivers
* are successfully probed.It's "need suppliers". As in, this is a device that "needs suppliers". So, no, this is not a list. This is a node in a list. And all "nodes in a list" are documented as "Hook" in rest of places in this file. So I think the documentation is correct as is.
Quoted 9 lines
Aha, I got confused about that while trying to keep everything straight.
Somehow I managed to conflate
needs_supplierswith the links between consumers and suppliers that are create viadevice_link_add().So original comment is fine.
It is getting late, so I'll continue with patches 2 and 3 tomorrow.
Thanks for the review.
-Saravana
Re: [PATCH v7 1/7] driver core: Add support for linking devices during device addition
Quoted 82 lines · Saravana Kannan
Quoted 73 lines · Frank Rowand
Quoted 31 lines · Saravana Kannan
Quoted 30 lines · Frank Rowand
Quoted 11 lines
Date: Tue, 23 Jul 2019 17:10:54 -0700
Subject: [PATCH v7 1/7] driver core: Add support for linking devices during
device addition
From: Saravana Kannan <[email protected]>When devices are added, the bus might want to create device links to track functional dependencies between supplier and consumer devices. This tracking of supplier-consumer relationship allows optimizing device probe order and tracking whether all consumers of a supplier are active. The
add_linksbus callback is added to support this.Change above to:
When devices are added, the bus may create device links to track which suppliers a consumer device depends upon. This
tracking of supplier-consumer relationship may be used to defer probing the driver of a consumer device before the driver(s) for its supplier device(s) are probed. It may also be used by a supplier driver to determine if all of its consumers have been successfully probed.
Theadd_linksbus callback is added to create the supplier device linksQuoted 5 lines
However, when consumer devices are added, they might not have a supplier device to link to despite needing mandatory resources/functionality from one or more suppliers. A
waiting_for_supplierslist is created to track such consumers and retry linking them when new devices get added.Change above to:
Quoted 39 lines
Quoted 5 lines
If a supplier device has not yet been created when the consumer device attempts to link it, the consumer device is added to the
wait_for_supplierslist. When supplier devices are created, the supplier device link will be added to the relevant consumer devices on thewait_for_supplierslist.I'll take these commit text suggestions if we decide to revert the entire series at the end of this review.
Quoted 29 lines
Quoted 21 lines
Signed-off-by Saravana Kannan <[email protected]>--- drivers/base/core.c | 83 ++++++++++++++++++++++++++++++++++++++++++ include/linux/device.h | 14 +++++++ 2 files changed, 97 insertions(+)drivers/base/core.c +2
-0@@ -44,6 +44,8 @@ early_param("sysfs.deprecated", sysfs_deprecated_setup);#endif /* Device links support. */ +static LIST_HEAD (wait_for_suppliers); +static DEFINE_MUTEX (wfs_lock);#ifdef CONFIG_SRCU static DEFINE_MUTEX (device_links_lock);drivers/base/core.c struct device_link *device_link_add(struct device *consumer, · line 408
Quoted 6 lines
drivers/base/core.c +2
-0@@ -401,6 +403,51 @@ struct device_link *device_link_add(struct device *consumer, }EXPORT_SYMBOL_GPL (device_link_add); +/** +* device_link_wait_for_supplier - Mark device as waiting for supplier *
device_link_wait_for_supplier- Add device towait_for_supplierslistQuoted 7 lines
As a meta-comment, I found this series very hard to understand in the context of reading the new code for the first time. When I read the code again in six months or a year or two years it will not be in near term memory and it will be as if I am reading it for the first time. A lot of my suggestions for changes of names are in that context -- the current names may be fine when one has recently read the code, but not so much when trying to read the whole thing again with a blank mind.
Quoted 7 lines
Thanks for the context.
Quoted 3 lines
The code also inherits a good deal of complexity because it does not stand alone in a nice discrete chunk, but instead delicately weaves into a more complex body of code.
I'll take this as a compliment :)
Please do!
Quoted 15 lines
Quoted 15 lines
When I was trying to understand the code, I wrote a lot of additional comments within my reply email to provide myself context, information about various things, and questions that I needed to answer (or if I could not answer to then ask you). Then I ended up being able to remove many of those notes before sending the reply.
Quoted 6 lines
I intentionally chose "Mark device..." because that's a better description of the semantics of the function instead of trying to describe the implementation. Whether I'm using a linked list or some other data structure should not be the one line documentation of a function. Unless the function is explicitly about operating on that specific data structure.
Quoted 8 lines
Quoted 5 lines
I agree with the intent of trying to describe the semantics of a function, especially at the API level where other systems (or drivers) would be using the function. But for this case the function is at the implementation level and describing explicitly what it is doing makes this much more readable for me.
Are you distinguishing between API level vs implementation level based on the function being "static"/not exported? I believe the earlier
No, being static helps say a function is not API, but an function that is not static may be intended to be used in a limited and constrained manner. I distinguished based on the usage of the function.
Quoted 69 lines
version of this series had this function as an exported API. So maybe that's why I had it as "Mark device".
Quoted 57 lines
I also find "Mark device" to be vague and not descriptive of what the intent is.
Quoted 47 lines
Quoted 36 lines
Quoted 25 lines
drivers/base/core.c +20
-0+* @consumer: Consumer device +* +* Marks the consumer device as waiting for suppliers to become available. The +* consumer device will never be probed until it's unmarked as waiting for +* suppliers. The caller is responsible for adding the link to the supplier +* once the supplier device is present. +* +* This function is NOT meant to be called from the probe function of the +* consumer but rather from code that creates/adds the consumer device. +*/ +static void device_link_wait_for_supplier (struct device *consumer) +{ +mutex_lock (&wfs_lock); +list_add_tail (&consumer->links.needs_suppliers, &wait_for_suppliers); +mutex_unlock (&wfs_lock); +} + +/** +* device_link_check_waiting_consumers - Try to remove from supplier wait list +* drivers/base/core.c struct device_link *device_link_add(struct device *consumer, · line 437
Quoted 10 lines
drivers/base/core.c +10
-0+* Loops through all consumers waiting on suppliers and tries to add all their +* supplier links. If that succeeds, the consumer device is unmarked as waiting +* for suppliers. Otherwise, they are left marked as waiting on suppliers, +* +* The add_links bus callback is expected to return 0 if it has found and added +* all the supplier links for the consumer device. It should return an error if +* it isn't able to do so. +* +* The caller of device_link_wait_for_supplier() is expected to call this once +* it's aware of potential suppliers becoming available. Quoted 6 lines
Change above comment to:
*
device_link_add_supplier_links- add links from consumer devices to
* supplier devices, leaving any consumer
* with inactive suppliers on the
*wait_for_supplierslistI didn't know that the first one line comment could span multiple lines. Good to know.
Quoted 6 lines
Quoted 1 line
* Scan all consumer devices in the devicetree.
This function doesn't have anything to do with devicetree. I've intentionally kept all OF related parts out of the driver/core because I hope that other busses can start using this feature too. So I can't take this bit.
Quoted 8 lines
My comment is left over from when I was taking notes, trying to understand the code.
At the moment, only devicetree is used as a source of the dependency information. The comment would better be re-phrased as:
* Scan all consumer devices in the firmware description of the hardware topology
Quoted 8 lines
Ok
Quoted 1 line
I did not ask why this feature is tied to _only_ the platform bus, but will now.
Because devicetree and platform bus the only ones I'm familiar with. If other busses want to add this, I'd be happy to help with code and/or direction/review. But I won't pretend to know anything about ACPI.
Sorry, you don't get to ignore other buses because you are not familiar with them.
I am not aware of any reason to exclude devices that on other buses and your answer below does not provide a valid technical reason why the new feature is correct when it excludes all other buses.
Quoted 7 lines
Quoted 3 lines
I do not know of any reason that a consumer / supplier relationship can not be between devices on different bus types. Do you know of such a reason?
Yes, it's hypothetically possible. But I haven't seen such a relationship being defined in DT. Nor somewhere else where this might be captured. So, how common/realistic is it?
It is entirely legal. I have no idea how common it is but that is not a valid reason to exclude other buses from the feature.
Quoted 18 lines
Quoted 18 lines
Quoted 11 lines
Quoted 10 lines
For any supplier device that
* is not already linked to the consumer device, add the supplier to the
* consumer device's device links.
*
* If all of a consumer device's suppliers are available then the consumer
* is removed from thewait_for_supplierslist (if previously on the list).
* Otherwise the consumer is added to thewait_for_supplierslist (if not
* already on the list).Quoted 1 line
Honestly, I don't think this is any better than what I already have.
Note that my version of these comments was written while I was reading the code, and did not have any big picture understanding yet. This will likely also be the mind set of most everyone who reads this code in the future, once it is woven into the kernel.
Quoted 41 lines
Quoted 2 lines
If you don't like the change, I can revisit it in a later version of the patch set.
I'll take in all the ones I feel are reasonable or don't feel strongly about. We can revisit the rest later.
Quoted 35 lines
Quoted 15 lines
Quoted 15 lines
* The
add_linksbus callback must return 0 if it has found and added all
* the supplier links for the consumer device. It must return an error if
* it is not able to do so.
*
* The caller ofdevice_link_wait_for_supplier()is expected to call this once
* it is aware of potential suppliers becoming available.drivers/base/core.c struct device_link *device_link_add(struct device *consumer, · line 442
Quoted 2 lines
drivers/base/core.c +2
-0+*/ +static void device_link_check_waiting_consumers (void )Function name is misleading and hides side effects.
Quoted 7 lines
Quoted 4 lines
I have not come up with a name that does not hide side effects, but a better name would be:
device_link_add_supplier_links()I kinda agree that it could afford a better name. The current name is too similar to
device_links_check_suppliers()and I never liked that.Naming new fields or variables related to device links looks pretty challenging to me, because of the desire to be part of device links and not a wart pasted on the side. So I share the pain in trying to find good names.
Quoted 2 lines
Maybe
device_link_add_missing_suppliers()?My first reaction was "yes, that sounds good". But then I stopped and tried to read the name out of context. The name is not adding the missing suppliers, it is saving the information that a supplier is not yet available (eg, is "missing"). I struggled in coming up with
Reading what you say below, and looking at the code again, what I say in that sentence is backwards. It is not adding the missing supplier device links, it is instead adding existing supplier device inks.
Quoted 3 lines
Quoted 1 line
the name that I suggested. We can keep thinking.
No, this function _IS_ about adding links to suppliers. These
You are mis-reading what I wrote. I said the function "is not adding the missing suppliers". You are converting that to "is not adding links to the missing suppliers".
My suggested name was hinting "add_supplier_links", which is what you say it does below. The name you suggest is hinting "add_missing_suppliers". Do you see the difference?
Quoted 19 lines
consumers were "saved" as "not yet having the supplier" earlier by
device_link_wait_for_supplier(). This function doesn't do that. This function is just trying to see if those missing suppliers are present now and if so adding a link to them from the "saved" consumers. I thinkdevice_link_add_missing_suppliers()is actually a pretty good name. Let me know what you think now.Quoted 12 lines
Quoted 4 lines
I don't think we need "links" repeated twice in the function name.
Yeah, I didn't like that either.
Quoted 3 lines
With this suggestion, what side effect is hidden in your opinion? That the fully linked consumer is removed from the "waiting for suppliers" list?
Quoted 6 lines
Quoted 2 lines
The side effect is that the function does not merely do a check. It also adds missing suppliers to a list.
No, it doesn't do that. I can't keep a list of things that aren't allocated yet :). In the whole patch series, we only keep a list of things (consumers) that are waiting on other things (missing suppliers).
OK, as I noted above, I stated that backwards. It is adding links for existing suppliers, not for the missing suppliers.
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
Maybe
device_link_try_removing_from_wfs()?I like that, other than the fact that it still does not provide a clue that the function is potentially adding suppliers to a list.
It doesn't. How would you add a supplier device to a list if the device itself isn't there? :)
Again, that should be existing suppliers, as you noted. But the point stands that the function is potentially adding links.
Quoted 224 lines
Quoted 127 lines
I think
part of the challenge is that the function does two things: (1) a check, and (2) potentially adding missing suppliers to a list. Maybe a simple one line comment at the call site, something like:/* adds missing suppliers to wfs */Quoted 15 lines
I'll wait for us to agree on a better name here before I change this.
Quoted 11 lines
drivers/base/core.c struct device_link *device_link_add(struct device *consumer, · line 447
Quoted 8 lines
drivers/base/core.c +8
-0+{ +struct device *dev, *tmp; + +mutex_lock (&wfs_lock); +list_for_each_entry_safe (dev, tmp, &wait_for_suppliers, + links.needs_suppliers) +if (!dev->bus->add_links (dev)) +list_del_init (&dev->links.needs_suppliers);Empties
dev->links.needs_suppliers, but does not remove dev fromwait_for_supplierslist. Where does that happen?Quoted 3 lines
I'll chalk this up to you having a long day or forgetting your coffee :)
list_del_init()does both of those things becauseneeds_suppliersis the node andwait_for_suppliersis the list.Yes, brain mis-fire on my part. I'll have to go back and look at the list related code again.
Quoted 21 lines
Quoted 21 lines
Quoted 9 lines
drivers/base/core.c +3
-0+mutex_unlock (&wfs_lock); +} +static void
device_link_free(struct device_link *link){ while (refcount_dec_not_one(&link->rpm_active))drivers/base/core.c int device_links_check_suppliers(struct device *dev) · line 590
Quoted 9 lines
drivers/base/core.c +6
-0@@ -535,6 +582,19 @@ int device_links_check_suppliers(struct device *dev)struct device_link *link;int ret =0 ; +/* +* If a device is waiting for one or more suppliers (in +* wait_for_suppliers list), it is not ready to probe yet. So just +* return -EPROBE_DEFER without having to check the links with existing +* suppliers. +*/ Change comment to:
Quoted 6 lines
Quoted 4 lines
/*
* Device waiting for supplier to become available is not allowed
* to probe
*/Po-tay-to. Po-tah-to? I think my comment is just as good.
If just as good and shorter, then better.
Also the original says "it is not ready to probe". That is not correct. It is ready to probe, it is just that the probe attempt will return -EPROBE_DEFER. Nit picky on my part, but tiny things like that mean I have to think harder. I have to think "why is it not ready to probe?". Maybe my version should have instead been something like:
* Device waiting for supplier to become available will return
* -EPROBE_DEFER if probed. Avoid the unneeded processing.Quoted 56 lines
Quoted 10 lines
drivers/base/core.c int device_links_check_suppliers(struct device *dev) · line 598
Quoted 9 lines
drivers/base/core.c +7
-0+mutex_lock (&wfs_lock); +if (!list_empty (&dev->links.needs_suppliers)) { +mutex_unlock (&wfs_lock); +return -EPROBE_DEFER ; + } +mutex_unlock (&wfs_lock); +device_links_write_lock ();Quoted 2 lines
Update
Documentation/driver-api/device_link.rstto reflect the check of&dev->links.needs_suppliersindevice_links_check_suppliers().Thanks! Will do.
Quoted 41 lines
Quoted 40 lines
drivers/base/core.c +18
-0list_for_each_entry (link, &dev->links.suppliers, c_node) {@@ -812,6 +872,10 @@ static void device_links_purge(struct device *dev) {struct device_link *link, *ln; +mutex_lock (&wfs_lock); +list_del (&dev->links.needs_suppliers); +mutex_unlock (&wfs_lock); +/* * Delete all of the remaining links from this device to any other * devices (either consumers or suppliers). @@ -1673,6 +1737,7 @@ void device_initialize(struct device *dev)#endif INIT_LIST_HEAD (&dev->links.consumers);INIT_LIST_HEAD (&dev->links.suppliers); +INIT_LIST_HEAD (&dev->links.needs_suppliers); dev->links.status =DL_DEV_NO_DRIVER ; }EXPORT_SYMBOL_GPL (device_initialize);@@ -2108,6 +2173,24 @@ int device_add(struct device *dev) BUS_NOTIFY_ADD_DEVICE, dev);kobject_uevent (&dev->kobj,KOBJ_ADD ); + +/* +* Check if any of the other devices (consumers) have been waiting for +* this device (supplier) to be added so that they can create a device +* link to it. +* +* This needs to happen after device_pm_add() because device_link_add() +* requires the supplier be registered before it's called. +* +* But this also needs to happe before bus_probe_device() to make sure +* waiting consumers can link to it before the driver is bound to the +* device and the driver sync_state callback is called for this device. +*/ Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
/*
* Add links to dev from any dependent consumer that has dev on it's
* list of needed suppliersThere is no list of needed suppliers.
"the other devices (consumers) have been waiting for this device (supplier)". Isn't that a list of needed suppliers?
No, that's a list of consumers that
needs_suppliers.Quoted 75 lines
Quoted 9 lines
Quoted 7 lines
(links.needs_suppliers).
Device_pm_add()
* must have previously registered dev to allow the links to be added. *
* The consumer links must be created before dev is probed because the
*sync_statecallback for dev will use the consumer links. */I think what I wrote is just as clear.
The original comment is vague. It does not explain why consumer links must be created before the probe. I had to go off and read other code to determine why that is true.
And again, brevity is better if otherwise just as clear.
Quoted 26 lines
Quoted 26 lines
Quoted 25 lines
drivers/base/core.c +5
-0+device_link_check_waiting_consumers (); + +if (dev->bus && dev->bus->add_links && dev->bus->add_links (dev)) +device_link_wait_for_supplier (dev); +bus_probe_device (dev);if (parent)klist_add_tail(&dev->p->knode_parent,include/linux/device.h +6
-0@@ -78,6 +78,17 @@ extern void bus_remove_file(struct bus_type *, struct bus_attribute *);* -EPROBE_DEFER it will queue the device for deferred probing. * @uevent: Called when a device is added, removed, or a few other things * that generate uevents to add the environment variables. +* @add_links: Called, perhaps multiple times per device, after a device is +* added to this bus. The function is expected to create device +* links to all the suppliers of the input device that are +* available at the time this function is called. As in, the +* function should NOT stop at the first failed device link if +* other unlinked supplier devices are present in the system. Quoted 4 lines
Quoted 1 line
* @add_links: Called after a device is added to this bus.
Why are you removing the "perhaps multiple times" part? that's true and that's how some of the other ops are documented.
I didn't remove it. I rephrased it with a little bit more explanation as "If some suppliers are not yet available, this function will be called again when the suppliers become available." (below).
Quoted 9 lines
Quoted 9 lines
The function is
* expected to create device links to all the suppliers of the * device that are available at the time this function is called. * The function must NOT stop at the first failed device link if * other unlinked supplier devices are present in the system. * If some suppliers are not yet available, this function will be * called again when the suppliers become available.Quoted 6 lines
Quoted 1 line
but
add_links()not needed, so moving this comment toof_link_to_suppliers()Sorry, I'm not sure I understand. Can you please explain what you are trying to say?
of_link_to_suppliers()is just one implementation ofadd_links(). The comment above is try for any bus trying to implementadd_links().This is conflating bus with the source of the firmware description of the hardware topology. For drivers that use various APIs to access firmware description of topology that may be either devicetree or ACPI the access is done via
fwnode_operations, based on struct device.fwnode (if I recall properly).Quoted 6 lines
I failed to completely address why
add_links()is not needed. The answer is that there should be a single function called for all buses. Then the proper firmware data source would be accessed via a structfwnode_operations.I think I left this out because I had not yet asked why this feature is tied only to the platform bus. Which I asked earlier in this reply.
Thanks for the pointer about fwnode and
fwnode_operations. I wasn't aware of those. I see where you are going with this. I see a couple of problems with this approach though:
Quoted 5 lines
1. How you interpret the properties of a fwnode is specific to the fw type. The clocks DT property isn't going to have the same definition in ACPI or some other firmware. Heck, I don't know if ACPI even has a clocks like property. So have one function to parse all the FW types doesn't make a lot of sense.
The functions in fwnode_operations are specific to the proper firmware. So there is a set of functions in a struct fwnode_operations for devicetree that only know about devicetree. And there is a different variable of type fwnode_operations that is initialized with ACPI specific functions.
Quoted 6 lines
2. If this common code is implemented as part of driver/base/, then at a minimum, I'll have to check if a fwnode is a DT node before I start interpreting the properties of a device's fwnode. But that means I'll have to include
linux/of.hto useis_of_node(). I don't like having driver/base code depend on OF or platform or ACPI headers.
You just use the function in the device's fwnode_operations (I think, I would have to go look at the precise way the code works because it has been quite a while since I've looked at it).
Quoted 3 lines
3. The supplier info doesn't always need to come from a firmware. So I don't want to limit it to that?
If you can find another source of topology info, then I would expect that another set of fwnode_operations functions would be created for the info source.
Quoted 6 lines
Also, I don't necessarily see this as conflating firmware (DT, ACPI, etc) with the bus (platform bus, ACPI bus, PCI bus). Whoever creates the device seems like the entity best suited to figure out the suppliers of the device (apart from the driver, obviously). So the bus deciding the suppliers doesn't seem wrong to me.
Patch 3 assigns the devicetree add_links function to the platform bus. It seems incorrect to me for of_platform_default_populate_init() to be changing a field in platform_bus_type.
of_platform_default_populate_init()
...
platform_bus_type.add_links = of_link_to_suppliers;Quoted 7 lines
In this specific case, I'm trying to address DT for now and leaving ACPI to whoever else wants to add device links based on ACPI data. Most OF/DT based devices end up in platform bus. So I'm just handling this in platform bus. If some other person wants this to work for ACPI bus or PCI bus, they are welcome to implement
add_links()for those busses? I'm nowhere close to an expert on those.
Devicetree is not limited to the platform bus.
Quoted 85 lines
Quoted 71 lines
Quoted 32 lines
Quoted 32 lines
include/linux/device.h extern void bus_remove_file(struct bus_type *, struct bus_attribute *); · line 91
Quoted 9 lines
include/linux/device.h +5
-0+* +* Return 0 if device links have been successfully created to all +* the suppliers of this device. Return an error if some of the +* suppliers are not yet available and this function needs to be +* reattempted in the future. *
* Return 0 if device links have been successfully created to all * the suppliers of this device. Return an error if some of the * suppliers are not yet available.include/linux/device.h struct bus_type { · line 138
Quoted 10 lines
include/linux/device.h +0
-0* @probe: Called when a new device or driver add to this bus, and callback * the specific driver's probe to initial the matched device. * @remove: Called when a device removed from this bus.
include/linux/device.h +1
-0@@ -122,6 +133,7 @@ struct bus_type {int (*match)(struct device *dev,struct device_driver *drv);int (*uevent)(struct device *dev,struct kobj_uevent_env *env); +int (*add_links)(struct device *dev);^^^^^^^^^ add_supplier ??? ^^^^^^^^^ add_suppliers ???^^^^^^^^^ link_suppliers ???Quoted 8 lines
Quoted 3 lines
^^^^^^^^^ add_supplier_dependency ??? ^^^^^^^^^ add_supplier_dependencies ???
add_links()not needed
add_links()was an intentional decision. There's no requirement that the bus should only create links from this device to its suppliers. If the bus also knows the consumers of this device (dev), then it can/should add those too.Is creating links to consumers of this device implemented in this patch series? If so, I overlooked it and will have to consider how that fits in to the design.
Quoted 17 lines
So, it shouldn't have "suppliers" in the
name.Quoted 14 lines
Quoted 13 lines
include/linux/device.h +0
-0int (*probe)(struct device *dev);int (*remove)(struct device *dev); void (*shutdown)(struct device *dev);include/linux/device.h +1
-0@@ -893,11 +905,13 @@ enum dl_dev_state {* struct dev_links_info - Device data related to device links. * @suppliers: List of links to supplier devices. * @consumers: List of links to consumer devices. +* @needs_suppliers: Hook to global list of devices waiting for suppliers. Quoted 7 lines
Quoted 2 lines
* @needs_suppliers: List of devices deferring probe until supplier drivers
* are successfully probed.It's "need suppliers". As in, this is a device that "needs suppliers". So, no, this is not a list. This is a node in a list. And all "nodes in a list" are documented as "Hook" in rest of places in this file. So I think the documentation is correct as is.
Quoted 9 lines
Aha, I got confused about that while trying to keep everything straight.
Somehow I managed to conflate
needs_supplierswith the links between consumers and suppliers that are create viadevice_link_add().So original comment is fine.
It is getting late, so I'll continue with patches 2 and 3 tomorrow.
Thanks for the review.
-Saravana
Re: [PATCH v7 1/7] driver core: Add support for linking devices during device addition
Quoted 197 lines · Frank Rowand
Quoted 82 lines · Saravana Kannan
Quoted 73 lines · Frank Rowand
Quoted 31 lines · Saravana Kannan
Quoted 30 lines · Frank Rowand
Quoted 11 lines
Date: Tue, 23 Jul 2019 17:10:54 -0700
Subject: [PATCH v7 1/7] driver core: Add support for linking devices during
device addition
From: Saravana Kannan <[email protected]>When devices are added, the bus might want to create device links to track functional dependencies between supplier and consumer devices. This tracking of supplier-consumer relationship allows optimizing device probe order and tracking whether all consumers of a supplier are active. The
add_linksbus callback is added to support this.Change above to:
When devices are added, the bus may create device links to track which suppliers a consumer device depends upon. This
tracking of supplier-consumer relationship may be used to defer probing the driver of a consumer device before the driver(s) for its supplier device(s) are probed. It may also be used by a supplier driver to determine if all of its consumers have been successfully probed.
Theadd_linksbus callback is added to create the supplier device linksQuoted 5 lines
However, when consumer devices are added, they might not have a supplier device to link to despite needing mandatory resources/functionality from one or more suppliers. A
waiting_for_supplierslist is created to track such consumers and retry linking them when new devices get added.Change above to:
Quoted 39 lines
Quoted 5 lines
If a supplier device has not yet been created when the consumer device attempts to link it, the consumer device is added to the
wait_for_supplierslist. When supplier devices are created, the supplier device link will be added to the relevant consumer devices on thewait_for_supplierslist.I'll take these commit text suggestions if we decide to revert the entire series at the end of this review.
Quoted 29 lines
Quoted 21 lines
Signed-off-by Saravana Kannan <[email protected]>--- drivers/base/core.c | 83 ++++++++++++++++++++++++++++++++++++++++++ include/linux/device.h | 14 +++++++ 2 files changed, 97 insertions(+)drivers/base/core.c +2
-0@@ -44,6 +44,8 @@ early_param("sysfs.deprecated", sysfs_deprecated_setup);#endif /* Device links support. */ +static LIST_HEAD (wait_for_suppliers); +static DEFINE_MUTEX (wfs_lock);#ifdef CONFIG_SRCU static DEFINE_MUTEX (device_links_lock);drivers/base/core.c struct device_link *device_link_add(struct device *consumer, · line 408
Quoted 6 lines
drivers/base/core.c +2
-0@@ -401,6 +403,51 @@ struct device_link *device_link_add(struct device *consumer, }EXPORT_SYMBOL_GPL (device_link_add); +/** +* device_link_wait_for_supplier - Mark device as waiting for supplier *
device_link_wait_for_supplier- Add device towait_for_supplierslistQuoted 7 lines
As a meta-comment, I found this series very hard to understand in the context of reading the new code for the first time. When I read the code again in six months or a year or two years it will not be in near term memory and it will be as if I am reading it for the first time. A lot of my suggestions for changes of names are in that context -- the current names may be fine when one has recently read the code, but not so much when trying to read the whole thing again with a blank mind.
Quoted 7 lines
Thanks for the context.
Quoted 3 lines
The code also inherits a good deal of complexity because it does not stand alone in a nice discrete chunk, but instead delicately weaves into a more complex body of code.
I'll take this as a compliment :)
Please do!
Quoted 15 lines
Quoted 15 lines
When I was trying to understand the code, I wrote a lot of additional comments within my reply email to provide myself context, information about various things, and questions that I needed to answer (or if I could not answer to then ask you). Then I ended up being able to remove many of those notes before sending the reply.
Quoted 6 lines
I intentionally chose "Mark device..." because that's a better description of the semantics of the function instead of trying to describe the implementation. Whether I'm using a linked list or some other data structure should not be the one line documentation of a function. Unless the function is explicitly about operating on that specific data structure.
Quoted 8 lines
Quoted 5 lines
I agree with the intent of trying to describe the semantics of a function, especially at the API level where other systems (or drivers) would be using the function. But for this case the function is at the implementation level and describing explicitly what it is doing makes this much more readable for me.
Are you distinguishing between API level vs implementation level based on the function being "static"/not exported? I believe the earlier
No, being static helps say a function is not API, but an function that is not static may be intended to be used in a limited and constrained manner. I distinguished based on the usage of the function.
Quoted 73 lines
version of this series had this function as an exported API. So maybe that's why I had it as "Mark device".
Quoted 57 lines
I also find "Mark device" to be vague and not descriptive of what the intent is.
Quoted 47 lines
Quoted 36 lines
Quoted 25 lines
drivers/base/core.c +20
-0+* @consumer: Consumer device +* +* Marks the consumer device as waiting for suppliers to become available. The +* consumer device will never be probed until it's unmarked as waiting for +* suppliers. The caller is responsible for adding the link to the supplier +* once the supplier device is present. +* +* This function is NOT meant to be called from the probe function of the +* consumer but rather from code that creates/adds the consumer device. +*/ +static void device_link_wait_for_supplier (struct device *consumer) +{ +mutex_lock (&wfs_lock); +list_add_tail (&consumer->links.needs_suppliers, &wait_for_suppliers); +mutex_unlock (&wfs_lock); +} + +/** +* device_link_check_waiting_consumers - Try to remove from supplier wait list +* drivers/base/core.c struct device_link *device_link_add(struct device *consumer, · line 437
Quoted 10 lines
drivers/base/core.c +10
-0+* Loops through all consumers waiting on suppliers and tries to add all their +* supplier links. If that succeeds, the consumer device is unmarked as waiting +* for suppliers. Otherwise, they are left marked as waiting on suppliers, +* +* The add_links bus callback is expected to return 0 if it has found and added +* all the supplier links for the consumer device. It should return an error if +* it isn't able to do so. +* +* The caller of device_link_wait_for_supplier() is expected to call this once +* it's aware of potential suppliers becoming available. Quoted 6 lines
Change above comment to:
*
device_link_add_supplier_links- add links from consumer devices to
* supplier devices, leaving any consumer
* with inactive suppliers on the
*wait_for_supplierslistI didn't know that the first one line comment could span multiple lines. Good to know.
Quoted 6 lines
Quoted 1 line
* Scan all consumer devices in the devicetree.
This function doesn't have anything to do with devicetree. I've intentionally kept all OF related parts out of the driver/core because I hope that other busses can start using this feature too. So I can't take this bit.
Quoted 8 lines
My comment is left over from when I was taking notes, trying to understand the code.
At the moment, only devicetree is used as a source of the dependency information. The comment would better be re-phrased as:
* Scan all consumer devices in the firmware description of the hardware topology
Ok
Quoted 1 line
I did not ask why this feature is tied to _only_ the platform bus, but will now.
Quoted 7 lines
Quoted 4 lines
Because devicetree and platform bus the only ones I'm familiar with. If other busses want to add this, I'd be happy to help with code and/or direction/review. But I won't pretend to know anything about ACPI.
Sorry, you don't get to ignore other buses because you are not familiar with them.
It's important that I don't design out other buses -- which I don't. But why would you want someone who has no idea of ACPI to write code for it? It's a futile effort that's going to be rejected by people who know ACPI anyway.
Quoted 7 lines
I am not aware of any reason to exclude devices that on other buses and your answer below does not provide a valid technical reason why the new feature is correct when it excludes all other buses.
Quoted 4 lines
Quoted 3 lines
I do not know of any reason that a consumer / supplier relationship can not be between devices on different bus types. Do you know of such a reason?
Quoted 6 lines
Quoted 3 lines
Yes, it's hypothetically possible. But I haven't seen such a relationship being defined in DT. Nor somewhere else where this might be captured. So, how common/realistic is it?
It is entirely legal. I have no idea how common it is but that is not a valid reason to exclude other buses from the feature.
I'm not going to write code for a hypothetical hardware scenario. Find one supported in upstream, show me that it'll benefit from this series and tell me how to interpret the dependency graph and then we'll talk about writing code for that.
Quoted 68 lines
Quoted 17 lines
Quoted 17 lines
Quoted 11 lines
Quoted 9 lines
For any supplier device that
* is not already linked to the consumer device, add the supplier to the
* consumer device's device links.
*
* If all of a consumer device's suppliers are available then the consumer
* is removed from thewait_for_supplierslist (if previously on the list).
* Otherwise the consumer is added to thewait_for_supplierslist (if not
* already on the list).Honestly, I don't think this is any better than what I already have.
Note that my version of these comments was written while I was reading the code, and did not have any big picture understanding yet. This will likely also be the mind set of most everyone who reads this code in the future, once it is woven into the kernel.
Quoted 41 lines
Quoted 2 lines
If you don't like the change, I can revisit it in a later version of the patch set.
I'll take in all the ones I feel are reasonable or don't feel strongly about. We can revisit the rest later.
Quoted 35 lines
Quoted 15 lines
Quoted 15 lines
* The
add_linksbus callback must return 0 if it has found and added all
* the supplier links for the consumer device. It must return an error if
* it is not able to do so.
*
* The caller ofdevice_link_wait_for_supplier()is expected to call this once
* it is aware of potential suppliers becoming available.drivers/base/core.c struct device_link *device_link_add(struct device *consumer, · line 442
Quoted 2 lines
drivers/base/core.c +2
-0+*/ +static void device_link_check_waiting_consumers (void )Function name is misleading and hides side effects.
Quoted 7 lines
Quoted 4 lines
I have not come up with a name that does not hide side effects, but a better name would be:
device_link_add_supplier_links()I kinda agree that it could afford a better name. The current name is too similar to
device_links_check_suppliers()and I never liked that.Naming new fields or variables related to device links looks pretty challenging to me, because of the desire to be part of device links and not a wart pasted on the side. So I share the pain in trying to find good names.
Quoted 2 lines
Maybe
device_link_add_missing_suppliers()?My first reaction was "yes, that sounds good". But then I stopped and tried to read the name out of context. The name is not adding the missing suppliers, it is saving the information that a supplier is not yet available (eg, is "missing"). I struggled in coming up with
Reading what you say below, and looking at the code again, what I say in that sentence is backwards. It is not adding the missing supplier device links, it is instead adding existing supplier device inks.
Quoted 3 lines
Quoted 1 line
the name that I suggested. We can keep thinking.
No, this function _IS_ about adding links to suppliers. These
Quoted 7 lines
You are mis-reading what I wrote. I said the function "is not adding the missing suppliers". You are converting that to "is not adding links to the missing suppliers".
My suggested name was hinting "
add_supplier_links", which is what you say it does below. The name you suggest is hinting "add_missing_suppliers". Do you see the difference?
Yeah, which is why I said earlier that I didn't want to repeat "links" twice in a function name. As in device_links_add_missing_supplier_links() has too many "links". In the context of device_links_, "add missing suppliers" means "add missing supplier links". Anyway, I think we can come back to figuring out a good name once we agree on the more important discussions further below.
Quoted 273 lines
Quoted 19 lines
consumers were "saved" as "not yet having the supplier" earlier by
device_link_wait_for_supplier(). This function doesn't do that. This function is just trying to see if those missing suppliers are present now and if so adding a link to them from the "saved" consumers. I thinkdevice_link_add_missing_suppliers()is actually a pretty good name. Let me know what you think now.Quoted 12 lines
Quoted 4 lines
I don't think we need "links" repeated twice in the function name.
Yeah, I didn't like that either.
Quoted 3 lines
With this suggestion, what side effect is hidden in your opinion? That the fully linked consumer is removed from the "waiting for suppliers" list?
Quoted 6 lines
Quoted 2 lines
The side effect is that the function does not merely do a check. It also adds missing suppliers to a list.
No, it doesn't do that. I can't keep a list of things that aren't allocated yet :). In the whole patch series, we only keep a list of things (consumers) that are waiting on other things (missing suppliers).
OK, as I noted above, I stated that backwards. It is adding links for existing suppliers, not for the missing suppliers.
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
Maybe
device_link_try_removing_from_wfs()?I like that, other than the fact that it still does not provide a clue that the function is potentially adding suppliers to a list.
It doesn't. How would you add a supplier device to a list if the device itself isn't there? :)
Again, that should be existing suppliers, as you noted. But the point stands that the function is potentially adding links.
Quoted 224 lines
Quoted 127 lines
I think
part of the challenge is that the function does two things: (1) a check, and (2) potentially adding missing suppliers to a list. Maybe a simple one line comment at the call site, something like:/* adds missing suppliers to wfs */Quoted 15 lines
I'll wait for us to agree on a better name here before I change this.
Quoted 11 lines
drivers/base/core.c struct device_link *device_link_add(struct device *consumer, · line 447
Quoted 8 lines
drivers/base/core.c +8
-0+{ +struct device *dev, *tmp; + +mutex_lock (&wfs_lock); +list_for_each_entry_safe (dev, tmp, &wait_for_suppliers, + links.needs_suppliers) +if (!dev->bus->add_links (dev)) +list_del_init (&dev->links.needs_suppliers);Empties
dev->links.needs_suppliers, but does not remove dev fromwait_for_supplierslist. Where does that happen?Quoted 3 lines
I'll chalk this up to you having a long day or forgetting your coffee :)
list_del_init()does both of those things becauseneeds_suppliersis the node andwait_for_suppliersis the list.Yes, brain mis-fire on my part. I'll have to go back and look at the list related code again.
Quoted 21 lines
Quoted 21 lines
Quoted 9 lines
drivers/base/core.c +3
-0+mutex_unlock (&wfs_lock); +} +static void
device_link_free(struct device_link *link){ while (refcount_dec_not_one(&link->rpm_active))drivers/base/core.c int device_links_check_suppliers(struct device *dev) · line 590
Quoted 9 lines
drivers/base/core.c +6
-0@@ -535,6 +582,19 @@ int device_links_check_suppliers(struct device *dev)struct device_link *link;int ret =0 ; +/* +* If a device is waiting for one or more suppliers (in +* wait_for_suppliers list), it is not ready to probe yet. So just +* return -EPROBE_DEFER without having to check the links with existing +* suppliers. +*/ Change comment to:
Quoted 6 lines
Quoted 4 lines
/*
* Device waiting for supplier to become available is not allowed
* to probe
*/Po-tay-to. Po-tah-to? I think my comment is just as good.
If just as good and shorter, then better.
Also the original says "it is not ready to probe". That is not correct. It is ready to probe, it is just that the probe attempt will return -EPROBE_DEFER. Nit picky on my part, but tiny things like that mean I have to think harder. I have to think "why is it not ready to probe?". Maybe my version should have instead been something like:
* Device waiting for supplier to become available will return
* -EPROBE_DEFER if probed. Avoid the unneeded processing.Quoted 56 lines
Quoted 10 lines
drivers/base/core.c int device_links_check_suppliers(struct device *dev) · line 598
Quoted 9 lines
drivers/base/core.c +7
-0+mutex_lock (&wfs_lock); +if (!list_empty (&dev->links.needs_suppliers)) { +mutex_unlock (&wfs_lock); +return -EPROBE_DEFER ; + } +mutex_unlock (&wfs_lock); +device_links_write_lock ();Quoted 2 lines
Update
Documentation/driver-api/device_link.rstto reflect the check of&dev->links.needs_suppliersindevice_links_check_suppliers().Thanks! Will do.
Quoted 41 lines
Quoted 40 lines
drivers/base/core.c +18
-0list_for_each_entry (link, &dev->links.suppliers, c_node) {@@ -812,6 +872,10 @@ static void device_links_purge(struct device *dev) {struct device_link *link, *ln; +mutex_lock (&wfs_lock); +list_del (&dev->links.needs_suppliers); +mutex_unlock (&wfs_lock); +/* * Delete all of the remaining links from this device to any other * devices (either consumers or suppliers). @@ -1673,6 +1737,7 @@ void device_initialize(struct device *dev)#endif INIT_LIST_HEAD (&dev->links.consumers);INIT_LIST_HEAD (&dev->links.suppliers); +INIT_LIST_HEAD (&dev->links.needs_suppliers); dev->links.status =DL_DEV_NO_DRIVER ; }EXPORT_SYMBOL_GPL (device_initialize);@@ -2108,6 +2173,24 @@ int device_add(struct device *dev) BUS_NOTIFY_ADD_DEVICE, dev);kobject_uevent (&dev->kobj,KOBJ_ADD ); + +/* +* Check if any of the other devices (consumers) have been waiting for +* this device (supplier) to be added so that they can create a device +* link to it. +* +* This needs to happen after device_pm_add() because device_link_add() +* requires the supplier be registered before it's called. +* +* But this also needs to happe before bus_probe_device() to make sure +* waiting consumers can link to it before the driver is bound to the +* device and the driver sync_state callback is called for this device. +*/ Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
/*
* Add links to dev from any dependent consumer that has dev on it's
* list of needed suppliersThere is no list of needed suppliers.
"the other devices (consumers) have been waiting for this device (supplier)". Isn't that a list of needed suppliers?
No, that's a list of consumers that
needs_suppliers.Quoted 75 lines
Quoted 9 lines
Quoted 7 lines
(links.needs_suppliers).
Device_pm_add()
* must have previously registered dev to allow the links to be added. *
* The consumer links must be created before dev is probed because the
*sync_statecallback for dev will use the consumer links. */I think what I wrote is just as clear.
The original comment is vague. It does not explain why consumer links must be created before the probe. I had to go off and read other code to determine why that is true.
And again, brevity is better if otherwise just as clear.
Quoted 26 lines
Quoted 26 lines
Quoted 25 lines
drivers/base/core.c +5
-0+device_link_check_waiting_consumers (); + +if (dev->bus && dev->bus->add_links && dev->bus->add_links (dev)) +device_link_wait_for_supplier (dev); +bus_probe_device (dev);if (parent)klist_add_tail(&dev->p->knode_parent,include/linux/device.h +6
-0@@ -78,6 +78,17 @@ extern void bus_remove_file(struct bus_type *, struct bus_attribute *);* -EPROBE_DEFER it will queue the device for deferred probing. * @uevent: Called when a device is added, removed, or a few other things * that generate uevents to add the environment variables. +* @add_links: Called, perhaps multiple times per device, after a device is +* added to this bus. The function is expected to create device +* links to all the suppliers of the input device that are +* available at the time this function is called. As in, the +* function should NOT stop at the first failed device link if +* other unlinked supplier devices are present in the system. Quoted 4 lines
Quoted 1 line
* @add_links: Called after a device is added to this bus.
Why are you removing the "perhaps multiple times" part? that's true and that's how some of the other ops are documented.
I didn't remove it. I rephrased it with a little bit more explanation as "If some suppliers are not yet available, this function will be called again when the suppliers become available." (below).
Quoted 9 lines
Quoted 9 lines
The function is
* expected to create device links to all the suppliers of the * device that are available at the time this function is called. * The function must NOT stop at the first failed device link if * other unlinked supplier devices are present in the system. * If some suppliers are not yet available, this function will be * called again when the suppliers become available.Quoted 6 lines
Quoted 1 line
but
add_links()not needed, so moving this comment toof_link_to_suppliers()Sorry, I'm not sure I understand. Can you please explain what you are trying to say?
of_link_to_suppliers()is just one implementation ofadd_links(). The comment above is try for any bus trying to implementadd_links().This is conflating bus with the source of the firmware description of the hardware topology. For drivers that use various APIs to access firmware description of topology that may be either devicetree or ACPI the access is done via
fwnode_operations, based on struct device.fwnode (if I recall properly).Quoted 6 lines
I failed to completely address why
add_links()is not needed. The answer is that there should be a single function called for all buses. Then the proper firmware data source would be accessed via a structfwnode_operations.I think I left this out because I had not yet asked why this feature is tied only to the platform bus. Which I asked earlier in this reply.
Thanks for the pointer about fwnode and
fwnode_operations. I wasn't aware of those. I see where you are going with this. I see a couple of problems with this approach though:Quoted 5 lines
1. How you interpret the properties of a fwnode is specific to the fw type. The clocks DT property isn't going to have the same definition in ACPI or some other firmware. Heck, I don't know if ACPI even has a clocks like property. So have one function to parse all the FW types doesn't make a lot of sense.
Quoted 5 lines
The functions in
fwnode_operationsare specific to the proper firmware. So there is a set of functions in a structfwnode_operationsfor devicetree that only know about devicetree. And there is a different variable of typefwnode_operationsthat is initialized with ACPI specific functions.
Yes, I understand how ops work :) So I have one ops (fwnode ops) to call that will read a property from DT or ACPI depending on where that specific device's firmware is from. But that's not my point here.
My point is that clock bindings in DT are under a "clocks" property that lists references (phandles) to the supplier. But in ACPI, the property might be called "clk" and could list references to actual clock IDs. So, you can't have one piece of code that works for all firmware even if I have one ops that can read properties from any firmware.
I'll still have to know what type the underlying firmware is before I try to interpret the properties. So having one function that parses DT and ACPI and whatever else would be a terrible and unnecessary design.
Quoted 9 lines
Quoted 5 lines
2. If this common code is implemented as part of driver/base/, then at a minimum, I'll have to check if a fwnode is a DT node before I start interpreting the properties of a device's fwnode. But that means I'll have to include
linux/of.hto useis_of_node(). I don't like having driver/base code depend on OF or platform or ACPI headers.You just use the function in the device's
fwnode_operations(I think, I would have to go look at the precise way the code works because it has been quite a while since I've looked at it).
Because you missed my point in (1) you are missing my point in (2). I'll wait for your updated reply.
Quoted 7 lines
Quoted 3 lines
3. The supplier info doesn't always need to come from a firmware. So I don't want to limit it to that?
If you can find another source of topology info, then I would expect that another set of
fwnode_operationsfunctions would be created for the info source.
The other source could just be C files in the kernel. Using fwnodes for that would be hacky. But let's sort (1) and (2) out first.
Quoted 7 lines
Quoted 6 lines
Also, I don't necessarily see this as conflating firmware (DT, ACPI, etc) with the bus (platform bus, ACPI bus, PCI bus). Whoever creates the device seems like the entity best suited to figure out the suppliers of the device (apart from the driver, obviously). So the bus deciding the suppliers doesn't seem wrong to me.
Quoted 9 lines
Patch 3 assigns the devicetree
add_linksfunction to the platform bus. It seems incorrect to me forof_platform_default_populate_init()to be changing a field inplatform_bus_type.of_platform_default_populate_init() ... platform_bus_type.add_links = of_link_to_suppliers;
I didn't want to have platform bus include OF header files.
Quoted 9 lines
Quoted 7 lines
In this specific case, I'm trying to address DT for now and leaving ACPI to whoever else wants to add device links based on ACPI data. Most OF/DT based devices end up in platform bus. So I'm just handling this in platform bus. If some other person wants this to work for ACPI bus or PCI bus, they are welcome to implement
add_links()for those busses? I'm nowhere close to an expert on those.Devicetree is not limited to the platform bus.
I know. That's why I said "most". PCI seems to have some DT support too.
Thanks,
Saravana
Re: [PATCH v7 1/7] driver core: Add support for linking devices during device addition
Quoted 203 lines · Saravana Kannan
Quoted 202 lines · Frank Rowand
Quoted 82 lines · Saravana Kannan
Quoted 73 lines · Frank Rowand
Quoted 31 lines · Saravana Kannan
Quoted 30 lines · Frank Rowand
Quoted 11 lines
Date: Tue, 23 Jul 2019 17:10:54 -0700
Subject: [PATCH v7 1/7] driver core: Add support for linking devices during
device addition
From: Saravana Kannan <[email protected]>When devices are added, the bus might want to create device links to track functional dependencies between supplier and consumer devices. This tracking of supplier-consumer relationship allows optimizing device probe order and tracking whether all consumers of a supplier are active. The
add_linksbus callback is added to support this.Change above to:
When devices are added, the bus may create device links to track which suppliers a consumer device depends upon. This
tracking of supplier-consumer relationship may be used to defer probing the driver of a consumer device before the driver(s) for its supplier device(s) are probed. It may also be used by a supplier driver to determine if all of its consumers have been successfully probed.
Theadd_linksbus callback is added to create the supplier device linksQuoted 5 lines
However, when consumer devices are added, they might not have a supplier device to link to despite needing mandatory resources/functionality from one or more suppliers. A
waiting_for_supplierslist is created to track such consumers and retry linking them when new devices get added.Change above to:
Quoted 39 lines
Quoted 5 lines
If a supplier device has not yet been created when the consumer device attempts to link it, the consumer device is added to the
wait_for_supplierslist. When supplier devices are created, the supplier device link will be added to the relevant consumer devices on thewait_for_supplierslist.I'll take these commit text suggestions if we decide to revert the entire series at the end of this review.
Quoted 29 lines
Quoted 21 lines
Signed-off-by Saravana Kannan <[email protected]>--- drivers/base/core.c | 83 ++++++++++++++++++++++++++++++++++++++++++ include/linux/device.h | 14 +++++++ 2 files changed, 97 insertions(+)drivers/base/core.c +2
-0@@ -44,6 +44,8 @@ early_param("sysfs.deprecated", sysfs_deprecated_setup);#endif /* Device links support. */ +static LIST_HEAD (wait_for_suppliers); +static DEFINE_MUTEX (wfs_lock);#ifdef CONFIG_SRCU static DEFINE_MUTEX (device_links_lock);drivers/base/core.c struct device_link *device_link_add(struct device *consumer, · line 408
Quoted 6 lines
drivers/base/core.c +2
-0@@ -401,6 +403,51 @@ struct device_link *device_link_add(struct device *consumer, }EXPORT_SYMBOL_GPL (device_link_add); +/** +* device_link_wait_for_supplier - Mark device as waiting for supplier *
device_link_wait_for_supplier- Add device towait_for_supplierslistQuoted 7 lines
As a meta-comment, I found this series very hard to understand in the context of reading the new code for the first time. When I read the code again in six months or a year or two years it will not be in near term memory and it will be as if I am reading it for the first time. A lot of my suggestions for changes of names are in that context -- the current names may be fine when one has recently read the code, but not so much when trying to read the whole thing again with a blank mind.
Quoted 7 lines
Thanks for the context.
Quoted 3 lines
The code also inherits a good deal of complexity because it does not stand alone in a nice discrete chunk, but instead delicately weaves into a more complex body of code.
I'll take this as a compliment :)
Please do!
Quoted 15 lines
Quoted 15 lines
When I was trying to understand the code, I wrote a lot of additional comments within my reply email to provide myself context, information about various things, and questions that I needed to answer (or if I could not answer to then ask you). Then I ended up being able to remove many of those notes before sending the reply.
Quoted 6 lines
I intentionally chose "Mark device..." because that's a better description of the semantics of the function instead of trying to describe the implementation. Whether I'm using a linked list or some other data structure should not be the one line documentation of a function. Unless the function is explicitly about operating on that specific data structure.
Quoted 8 lines
Quoted 5 lines
I agree with the intent of trying to describe the semantics of a function, especially at the API level where other systems (or drivers) would be using the function. But for this case the function is at the implementation level and describing explicitly what it is doing makes this much more readable for me.
Are you distinguishing between API level vs implementation level based on the function being "static"/not exported? I believe the earlier
No, being static helps say a function is not API, but an function that is not static may be intended to be used in a limited and constrained manner. I distinguished based on the usage of the function.
Quoted 77 lines
version of this series had this function as an exported API. So maybe that's why I had it as "Mark device".
Quoted 57 lines
I also find "Mark device" to be vague and not descriptive of what the intent is.
Quoted 47 lines
Quoted 36 lines
Quoted 25 lines
drivers/base/core.c +20
-0+* @consumer: Consumer device +* +* Marks the consumer device as waiting for suppliers to become available. The +* consumer device will never be probed until it's unmarked as waiting for +* suppliers. The caller is responsible for adding the link to the supplier +* once the supplier device is present. +* +* This function is NOT meant to be called from the probe function of the +* consumer but rather from code that creates/adds the consumer device. +*/ +static void device_link_wait_for_supplier (struct device *consumer) +{ +mutex_lock (&wfs_lock); +list_add_tail (&consumer->links.needs_suppliers, &wait_for_suppliers); +mutex_unlock (&wfs_lock); +} + +/** +* device_link_check_waiting_consumers - Try to remove from supplier wait list +* drivers/base/core.c struct device_link *device_link_add(struct device *consumer, · line 440
Quoted 10 lines
drivers/base/core.c +10
-0+* Loops through all consumers waiting on suppliers and tries to add all their +* supplier links. If that succeeds, the consumer device is unmarked as waiting +* for suppliers. Otherwise, they are left marked as waiting on suppliers, +* +* The add_links bus callback is expected to return 0 if it has found and added +* all the supplier links for the consumer device. It should return an error if +* it isn't able to do so. +* +* The caller of device_link_wait_for_supplier() is expected to call this once +* it's aware of potential suppliers becoming available. Quoted 6 lines
Change above comment to:
*
device_link_add_supplier_links- add links from consumer devices to
* supplier devices, leaving any consumer
* with inactive suppliers on the
*wait_for_supplierslistI didn't know that the first one line comment could span multiple lines. Good to know.
Quoted 6 lines
Quoted 1 line
* Scan all consumer devices in the devicetree.
This function doesn't have anything to do with devicetree. I've intentionally kept all OF related parts out of the driver/core because I hope that other busses can start using this feature too. So I can't take this bit.
Quoted 8 lines
My comment is left over from when I was taking notes, trying to understand the code.
At the moment, only devicetree is used as a source of the dependency information. The comment would better be re-phrased as:
* Scan all consumer devices in the firmware description of the hardware topology
Ok
Quoted 1 line
I did not ask why this feature is tied to _only_ the platform bus, but will now.
Because devicetree and platform bus the only ones I'm familiar with. If other busses want to add this, I'd be happy to help with code and/or direction/review. But I won't pretend to know anything about ACPI.
Quoted 7 lines
Quoted 2 lines
Sorry, you don't get to ignore other buses because you are not familiar with them.
It's important that I don't design out other buses -- which I don't. But why would you want someone who has no idea of ACPI to write code for it? It's a futile effort that's going to be rejected by people who know ACPI anyway.
ACPI is not a bus.
Devicetree is not a bus.
A devicetree can contain multiple buses in the topology that is described.
Quoted 12 lines
Quoted 12 lines
I am not aware of any reason to exclude devices that on other buses and your answer below does not provide a valid technical reason why the new feature is correct when it excludes all other buses.
Quoted 7 lines
Quoted 3 lines
I do not know of any reason that a consumer / supplier relationship can not be between devices on different bus types. Do you know of such a reason?
Yes, it's hypothetically possible. But I haven't seen such a relationship being defined in DT. Nor somewhere else where this might be captured. So, how common/realistic is it?
Quoted 7 lines
Quoted 2 lines
It is entirely legal. I have no idea how common it is but that is not a valid reason to exclude other buses from the feature.
I'm not going to write code for a hypothetical hardware scenario. Find one supported in upstream, show me that it'll benefit from this series and tell me how to interpret the dependency graph and then we'll talk about writing code for that.
You don't get to implement a general feature in a way that only supports a subset of potential devicetree users. Note the word "general". This is not a small isolated feature.
Now, am I being inconsistent if I say that it is ok for the feature to only support devicetree systems, or only support ACPI systems? I'll have to ponder that.
But I don't think the question of only platform buses or all buses needs to be resolved because I don't think that the add_links function is a bus specific function. The add_links function is specific to devicetree or ACPI.
We seem to be talking past each other on this point right now. I don't now how to get our minds to the same place, but let's keep trying.
Quoted 77 lines
Quoted 69 lines
Quoted 18 lines
Quoted 18 lines
Quoted 11 lines
Quoted 10 lines
For any supplier device that
* is not already linked to the consumer device, add the supplier to the
* consumer device's device links.
*
* If all of a consumer device's suppliers are available then the consumer
* is removed from thewait_for_supplierslist (if previously on the list).
* Otherwise the consumer is added to thewait_for_supplierslist (if not
* already on the list).Quoted 1 line
Honestly, I don't think this is any better than what I already have.
Note that my version of these comments was written while I was reading the code, and did not have any big picture understanding yet. This will likely also be the mind set of most everyone who reads this code in the future, once it is woven into the kernel.
Quoted 41 lines
Quoted 2 lines
If you don't like the change, I can revisit it in a later version of the patch set.
I'll take in all the ones I feel are reasonable or don't feel strongly about. We can revisit the rest later.
Quoted 35 lines
Quoted 15 lines
Quoted 15 lines
* The
add_linksbus callback must return 0 if it has found and added all
* the supplier links for the consumer device. It must return an error if
* it is not able to do so.
*
* The caller ofdevice_link_wait_for_supplier()is expected to call this once
* it is aware of potential suppliers becoming available.drivers/base/core.c struct device_link *device_link_add(struct device *consumer, · line 442
Quoted 2 lines
drivers/base/core.c +2
-0+*/ +static void device_link_check_waiting_consumers (void )Function name is misleading and hides side effects.
Quoted 7 lines
Quoted 4 lines
I have not come up with a name that does not hide side effects, but a better name would be:
device_link_add_supplier_links()I kinda agree that it could afford a better name. The current name is too similar to
device_links_check_suppliers()and I never liked that.Naming new fields or variables related to device links looks pretty challenging to me, because of the desire to be part of device links and not a wart pasted on the side. So I share the pain in trying to find good names.
Quoted 2 lines
Maybe
device_link_add_missing_suppliers()?My first reaction was "yes, that sounds good". But then I stopped and tried to read the name out of context. The name is not adding the missing suppliers, it is saving the information that a supplier is not yet available (eg, is "missing"). I struggled in coming up with
Reading what you say below, and looking at the code again, what I say in that sentence is backwards. It is not adding the missing supplier device links, it is instead adding existing supplier device inks.
Quoted 3 lines
Quoted 1 line
the name that I suggested. We can keep thinking.
No, this function _IS_ about adding links to suppliers. These
Quoted 7 lines
You are mis-reading what I wrote. I said the function "is not adding the missing suppliers". You are converting that to "is not adding links to the missing suppliers".
My suggested name was hinting "
add_supplier_links", which is what you say it does below. The name you suggest is hinting "add_missing_suppliers". Do you see the difference?
Quoted 7 lines
Yeah, which is why I said earlier that I didn't want to repeat "links" twice in a function name. As in
device_links_add_missing_supplier_links()has too many "links". In the context ofdevice_links_, "add missing suppliers" means "add missing supplier links". Anyway, I think we can come back to figuring out a good name once we agree on the more important discussions further below.
Yes, later is fine. This is a detail.
Quoted 291 lines
Quoted 274 lines
Quoted 20 lines
consumers were "saved" as "not yet having the supplier" earlier by
device_link_wait_for_supplier(). This function doesn't do that. This function is just trying to see if those missing suppliers are present now and if so adding a link to them from the "saved" consumers. I thinkdevice_link_add_missing_suppliers()is actually a pretty good name. Let me know what you think now.Quoted 12 lines
Quoted 4 lines
I don't think we need "links" repeated twice in the function name.
Yeah, I didn't like that either.
Quoted 3 lines
With this suggestion, what side effect is hidden in your opinion? That the fully linked consumer is removed from the "waiting for suppliers" list?
Quoted 6 lines
Quoted 2 lines
The side effect is that the function does not merely do a check. It also adds missing suppliers to a list.
No, it doesn't do that. I can't keep a list of things that aren't allocated yet :). In the whole patch series, we only keep a list of things (consumers) that are waiting on other things (missing suppliers).
OK, as I noted above, I stated that backwards. It is adding links for existing suppliers, not for the missing suppliers.
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
Maybe
device_link_try_removing_from_wfs()?I like that, other than the fact that it still does not provide a clue that the function is potentially adding suppliers to a list.
It doesn't. How would you add a supplier device to a list if the device itself isn't there? :)
Again, that should be existing suppliers, as you noted. But the point stands that the function is potentially adding links.
Quoted 224 lines
Quoted 127 lines
I think
part of the challenge is that the function does two things: (1) a check, and (2) potentially adding missing suppliers to a list. Maybe a simple one line comment at the call site, something like:/* adds missing suppliers to wfs */Quoted 15 lines
I'll wait for us to agree on a better name here before I change this.
Quoted 11 lines
drivers/base/core.c struct device_link *device_link_add(struct device *consumer, · line 450
Quoted 8 lines
drivers/base/core.c +8
-0+{ +struct device *dev, *tmp; + +mutex_lock (&wfs_lock); +list_for_each_entry_safe (dev, tmp, &wait_for_suppliers, + links.needs_suppliers) +if (!dev->bus->add_links (dev)) +list_del_init (&dev->links.needs_suppliers);Empties
dev->links.needs_suppliers, but does not remove dev fromwait_for_supplierslist. Where does that happen?Quoted 3 lines
I'll chalk this up to you having a long day or forgetting your coffee :)
list_del_init()does both of those things becauseneeds_suppliersis the node andwait_for_suppliersis the list.Yes, brain mis-fire on my part. I'll have to go back and look at the list related code again.
Quoted 21 lines
Quoted 21 lines
Quoted 9 lines
drivers/base/core.c +3
-0+mutex_unlock (&wfs_lock); +} +static void
device_link_free(struct device_link *link){ while (refcount_dec_not_one(&link->rpm_active))drivers/base/core.c int device_links_check_suppliers(struct device *dev) · line 590
Quoted 9 lines
drivers/base/core.c +6
-0@@ -535,6 +582,19 @@ int device_links_check_suppliers(struct device *dev)struct device_link *link;int ret =0 ; +/* +* If a device is waiting for one or more suppliers (in +* wait_for_suppliers list), it is not ready to probe yet. So just +* return -EPROBE_DEFER without having to check the links with existing +* suppliers. +*/ Change comment to:
Quoted 6 lines
Quoted 4 lines
/*
* Device waiting for supplier to become available is not allowed
* to probe
*/Po-tay-to. Po-tah-to? I think my comment is just as good.
If just as good and shorter, then better.
Also the original says "it is not ready to probe". That is not correct. It is ready to probe, it is just that the probe attempt will return -EPROBE_DEFER. Nit picky on my part, but tiny things like that mean I have to think harder. I have to think "why is it not ready to probe?". Maybe my version should have instead been something like:
* Device waiting for supplier to become available will return
* -EPROBE_DEFER if probed. Avoid the unneeded processing.Quoted 56 lines
Quoted 10 lines
drivers/base/core.c int device_links_check_suppliers(struct device *dev) · line 598
Quoted 9 lines
drivers/base/core.c +7
-0+mutex_lock (&wfs_lock); +if (!list_empty (&dev->links.needs_suppliers)) { +mutex_unlock (&wfs_lock); +return -EPROBE_DEFER ; + } +mutex_unlock (&wfs_lock); +device_links_write_lock ();Quoted 2 lines
Update
Documentation/driver-api/device_link.rstto reflect the check of&dev->links.needs_suppliersindevice_links_check_suppliers().Thanks! Will do.
Quoted 41 lines
Quoted 40 lines
drivers/base/core.c +18
-0list_for_each_entry (link, &dev->links.suppliers, c_node) {@@ -812,6 +872,10 @@ static void device_links_purge(struct device *dev) {struct device_link *link, *ln; +mutex_lock (&wfs_lock); +list_del (&dev->links.needs_suppliers); +mutex_unlock (&wfs_lock); +/* * Delete all of the remaining links from this device to any other * devices (either consumers or suppliers). @@ -1673,6 +1737,7 @@ void device_initialize(struct device *dev)#endif INIT_LIST_HEAD (&dev->links.consumers);INIT_LIST_HEAD (&dev->links.suppliers); +INIT_LIST_HEAD (&dev->links.needs_suppliers); dev->links.status =DL_DEV_NO_DRIVER ; }EXPORT_SYMBOL_GPL (device_initialize);@@ -2108,6 +2173,24 @@ int device_add(struct device *dev) BUS_NOTIFY_ADD_DEVICE, dev);kobject_uevent (&dev->kobj,KOBJ_ADD ); + +/* +* Check if any of the other devices (consumers) have been waiting for +* this device (supplier) to be added so that they can create a device +* link to it. +* +* This needs to happen after device_pm_add() because device_link_add() +* requires the supplier be registered before it's called. +* +* But this also needs to happe before bus_probe_device() to make sure +* waiting consumers can link to it before the driver is bound to the +* device and the driver sync_state callback is called for this device. +*/ Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
/*
* Add links to dev from any dependent consumer that has dev on it's
* list of needed suppliersThere is no list of needed suppliers.
"the other devices (consumers) have been waiting for this device (supplier)". Isn't that a list of needed suppliers?
No, that's a list of consumers that
needs_suppliers.Quoted 75 lines
Quoted 9 lines
Quoted 7 lines
(links.needs_suppliers).
Device_pm_add()
* must have previously registered dev to allow the links to be added. *
* The consumer links must be created before dev is probed because the
*sync_statecallback for dev will use the consumer links. */I think what I wrote is just as clear.
The original comment is vague. It does not explain why consumer links must be created before the probe. I had to go off and read other code to determine why that is true.
And again, brevity is better if otherwise just as clear.
Quoted 26 lines
Quoted 26 lines
Quoted 25 lines
drivers/base/core.c +5
-0+device_link_check_waiting_consumers (); + +if (dev->bus && dev->bus->add_links && dev->bus->add_links (dev)) +device_link_wait_for_supplier (dev); +bus_probe_device (dev);if (parent)klist_add_tail(&dev->p->knode_parent,include/linux/device.h +6
-0@@ -78,6 +78,17 @@ extern void bus_remove_file(struct bus_type *, struct bus_attribute *);* -EPROBE_DEFER it will queue the device for deferred probing. * @uevent: Called when a device is added, removed, or a few other things * that generate uevents to add the environment variables. +* @add_links: Called, perhaps multiple times per device, after a device is +* added to this bus. The function is expected to create device +* links to all the suppliers of the input device that are +* available at the time this function is called. As in, the +* function should NOT stop at the first failed device link if +* other unlinked supplier devices are present in the system. Quoted 4 lines
Quoted 1 line
* @add_links: Called after a device is added to this bus.
Why are you removing the "perhaps multiple times" part? that's true and that's how some of the other ops are documented.
I didn't remove it. I rephrased it with a little bit more explanation as "If some suppliers are not yet available, this function will be called again when the suppliers become available." (below).
Quoted 9 lines
Quoted 9 lines
The function is
* expected to create device links to all the suppliers of the * device that are available at the time this function is called. * The function must NOT stop at the first failed device link if * other unlinked supplier devices are present in the system. * If some suppliers are not yet available, this function will be * called again when the suppliers become available.Quoted 6 lines
Quoted 1 line
but
add_links()not needed, so moving this comment toof_link_to_suppliers()Sorry, I'm not sure I understand. Can you please explain what you are trying to say?
of_link_to_suppliers()is just one implementation ofadd_links(). The comment above is try for any bus trying to implementadd_links().This is conflating bus with the source of the firmware description of the hardware topology. For drivers that use various APIs to access firmware description of topology that may be either devicetree or ACPI the access is done via
fwnode_operations, based on struct device.fwnode (if I recall properly).Quoted 6 lines
I failed to completely address why
add_links()is not needed. The answer is that there should be a single function called for all buses. Then the proper firmware data source would be accessed via a structfwnode_operations.I think I left this out because I had not yet asked why this feature is tied only to the platform bus. Which I asked earlier in this reply.
Thanks for the pointer about fwnode and
fwnode_operations. I wasn't aware of those. I see where you are going with this. I see a couple of problems with this approach though:Quoted 5 lines
1. How you interpret the properties of a fwnode is specific to the fw type. The clocks DT property isn't going to have the same definition in ACPI or some other firmware. Heck, I don't know if ACPI even has a clocks like property. So have one function to parse all the FW types doesn't make a lot of sense.
Quoted 5 lines
The functions in
fwnode_operationsare specific to the proper firmware. So there is a set of functions in a structfwnode_operationsfor devicetree that only know about devicetree. And there is a different variable of typefwnode_operationsthat is initialized with ACPI specific functions.Yes, I understand how ops work :) So I have one ops (fwnode ops) to call that will read a property from DT or ACPI depending on where that specific device's firmware is from. But that's not my point here.
My point is that clock bindings in DT are under a "clocks" property that lists references (phandles) to the supplier. But in ACPI, the property might be called "clk" and could list references to actual clock IDs. So, you can't have one piece of code that works for all firmware even if I have one ops that can read properties from any firmware.
Quoted 3 lines
I'll still have to know what type the underlying firmware is before I try to interpret the properties. So having one function that parses DT and ACPI and whatever else would be a terrible and unnecessary design.
You have already implemented the devicetree function, which is of_link_to_suppliers(). The devicetree fwnode_operations would have a pointer to of_link_to_suppliers().
If ACPI support is added, there would be an analogous ACPI aware function that would essentially do the same thing that of_link_to_suppliers() does. This would be in the ACPI version of fwnode_operations.
There would not be a single function that is both devicetree aware and ACPI aware.
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
2. If this common code is implemented as part of driver/base/, then at a minimum, I'll have to check if a fwnode is a DT node before I start interpreting the properties of a device's fwnode. But that means I'll have to include
linux/of.hto useis_of_node(). I don't like having driver/base code depend on OF or platform or ACPI headers.
Quoted 6 lines
Quoted 3 lines
You just use the function in the device's
fwnode_operations(I think, I would have to go look at the precise way the code works because it has been quite a while since I've looked at it).Because you missed my point in (1) you are missing my point in (2). I'll wait for your updated reply.
We are still talking at cross purposes. If my reply to (1) does not change that, I'll have to go dig into how the fwnode framework figures out which set of fwnode_operations to use for each device.
Quoted 11 lines
Quoted 8 lines
Quoted 4 lines
3. The supplier info doesn't always need to come from a firmware. So I don't want to limit it to that?
If you can find another source of topology info, then I would expect that another set of
fwnode_operationsfunctions would be created for the info source.The other source could just be C files in the kernel. Using fwnodes for that would be hacky. But let's sort (1) and (2) out first.
I suspected that might be the other source. But we have been trying to deprecate this type of data.
Quoted 13 lines
Quoted 13 lines
Quoted 7 lines
Also, I don't necessarily see this as conflating firmware (DT, ACPI, etc) with the bus (platform bus, ACPI bus, PCI bus). Whoever creates the device seems like the entity best suited to figure out the suppliers of the device (apart from the driver, obviously). So the bus deciding the suppliers doesn't seem wrong to me.
Patch 3 assigns the devicetree
add_linksfunction to the platform bus. It seems incorrect to me forof_platform_default_populate_init()to be changing a field inplatform_bus_type.
Quoted 6 lines
Quoted 4 lines
of_platform_default_populate_init() ... platform_bus_type.add_links = of_link_to_suppliers;I didn't want to have platform bus include OF header files.
Yet another clue that the function pointer should not belong to the bus.
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
In this specific case, I'm trying to address DT for now and leaving ACPI to whoever else wants to add device links based on ACPI data. Most OF/DT based devices end up in platform bus. So I'm just handling this in platform bus. If some other person wants this to work for ACPI bus or PCI bus, they are welcome to implement
add_links()for those busses? I'm nowhere close to an expert on those.
Quoted 7 lines
Quoted 1 line
Devicetree is not limited to the platform bus.
I know. That's why I said "most". PCI seems to have some DT support too.
Thanks,
Saravana
-Frank
Greg Kroah-Hartman
maintainerRe: [PATCH v7 1/7] driver core: Add support for linking devices during device addition
Quoted 6 lines · Frank Rowand
Quoted 5 lines · Saravana Kannan
Quoted 4 lines · Frank Rowand
Quoted 2 lines · Saravana Kannan
Quoted 1 line · Frank Rowand
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines · Saravana Kannan
Quoted 6 lines · Frank Rowand
Quoted 5 lines
Quoted 5 lines
Date: Tue, 23 Jul 2019 17:10:54 -0700
Subject: [PATCH v7 1/7] driver core: Add support for linking devices during
device addition
From: Saravana Kannan <[email protected]>
This is a "fun" thread :(
You two should get together in person this week and talk. I think you both will be at ELC, can we do this tomorrow or Thursday so we can hash it out in a way that doesn't end up talking past each other, like I feel is happening here right now?
thanks,
greg k-h
Re: [PATCH v7 1/7] driver core: Add support for linking devices during device addition
Quoted 19 lines · Greg Kroah-Hartman
Quoted 2 lines · Frank Rowand
Quoted 1 line · Saravana Kannan
On Mon, Aug 19, 2019 at 9:25 PM Frank Rowand <[email protected]>
Quoted 3 lines · wrote:
Quoted 3 lines
Quoted 3 lines · Saravana Kannan
Quoted 1 line
On Sun, Aug 18, 2019 at 8:38 PM Frank Rowand <[email protected]>
Quoted 3 lines · wrote:
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines · Saravana Kannan
Quoted 1 line
On Wed, Aug 7, 2019 at 7:04 PM Frank Rowand <
Quoted 3 lines · [email protected]> wrote:
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Date: Tue, 23 Jul 2019 17:10:54 -0700
Subject: [PATCH v7 1/7] driver core: Add support for linkingdevices during
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
device addition
From: Saravana Kannan <[email protected]>
Quoted 7 lines
This is a "fun" thread :(
You two should get together in person this week and talk. I think you both will be at ELC, can we do this tomorrow or Thursday so we can hash it out in a way that doesn't end up talking past each other, like I feel is happening here right now?
That would be great. Wednesday would be better for me. I might not make it to ELC on Thursday. Let us know Frank.
Thanks,
Saravana
Re: [PATCH v7 1/7] driver core: Add support for linking devices during device addition
Quoted 27 lines · Saravana Kannan
On Tue, Aug 20, 2019, 6:56 PM Greg Kroah-Hartman <[email protected] <mailto:[email protected]>> wrote:
On Tue, Aug 20, 2019 at 06:06:55PM -0700, Frank Rowand wrote: > On 8/20/19 3:10 PM, Saravana Kannan wrote: > > On Mon, Aug 19, 2019 at 9:25 PM Frank Rowand <[email protected] <mailto:[email protected]>> wrote: > >> > >> On 8/19/19 5:00 PM, Saravana Kannan wrote: > >>> On Sun, Aug 18, 2019 at 8:38 PM Frank Rowand <[email protected] <mailto:[email protected]>> wrote: > >>>> > >>>> On 8/15/19 6:50 PM, Saravana Kannan wrote: > >>>>> On Wed, Aug 7, 2019 at 7:04 PM Frank Rowand <[email protected] <mailto:[email protected]>> wrote: > >>>>>> > >>>>>>> Date: Tue, 23 Jul 2019 17:10:54 -0700 > >>>>>>> Subject: [PATCH v7 1/7] driver core: Add support for linking devices during > >>>>>>> device addition > >>>>>>> From: Saravana Kannan <[email protected] <mailto:[email protected]>> This is a "fun" thread :( You two should get together in person this week and talk. I think you both will be at ELC, can we do this tomorrow or Thursday so we can hash it out in a way that doesn't end up talking past each other, like I feel is happening here right now?
Quoted 4 lines
That would be great. Wednesday would be better for me. I might not make it to ELC on Thursday. Let us know Frank.
Thanks,
Saravana
I am really glad that you are here at ELC. It should be very productive to sit down together and figure some things out.
I'll send a separate reply with my phone number off list.
-Frank
Re: [PATCH v7 1/7] driver core: Add support for linking devices during device addition
In reply to Greg Kroah-HartmanQuoted 16 lines · Greg Kroah-Hartman
Quoted 6 lines · Frank Rowand
Quoted 5 lines · Saravana Kannan
Quoted 4 lines · Frank Rowand
Quoted 2 lines · Saravana Kannan
Quoted 1 line · Frank Rowand
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines · Saravana Kannan
Quoted 6 lines · Frank Rowand
Quoted 5 lines
Quoted 5 lines
Date: Tue, 23 Jul 2019 17:10:54 -0700
Subject: [PATCH v7 1/7] driver core: Add support for linking devices during
device addition
From: Saravana Kannan <[email protected]>
Quoted 7 lines
This is a "fun" thread :(
You two should get together in person this week and talk. I think you both will be at ELC, can we do this tomorrow or Thursday so we can hash it out in a way that doesn't end up talking past each other, like I feel is happening here right now?
Resending again due to HTML (Sorry, was a mobile reply)
That would be great. Wednesday would be better for me. I might not make it to ELC on Thursday. Let us know Frank.
Thanks,
Saravana
Re: [PATCH v7 1/7] driver core: Add support for linking devices during device addition
In reply to Frank RowandQuoted 213 lines · Frank Rowand
Quoted 203 lines · Saravana Kannan
Quoted 202 lines · Frank Rowand
Quoted 82 lines · Saravana Kannan
Quoted 73 lines · Frank Rowand
Quoted 31 lines · Saravana Kannan
Quoted 30 lines · Frank Rowand
Quoted 11 lines
Date: Tue, 23 Jul 2019 17:10:54 -0700
Subject: [PATCH v7 1/7] driver core: Add support for linking devices during
device addition
From: Saravana Kannan <[email protected]>When devices are added, the bus might want to create device links to track functional dependencies between supplier and consumer devices. This tracking of supplier-consumer relationship allows optimizing device probe order and tracking whether all consumers of a supplier are active. The
add_linksbus callback is added to support this.Change above to:
When devices are added, the bus may create device links to track which suppliers a consumer device depends upon. This
tracking of supplier-consumer relationship may be used to defer probing the driver of a consumer device before the driver(s) for its supplier device(s) are probed. It may also be used by a supplier driver to determine if all of its consumers have been successfully probed.
Theadd_linksbus callback is added to create the supplier device linksQuoted 5 lines
However, when consumer devices are added, they might not have a supplier device to link to despite needing mandatory resources/functionality from one or more suppliers. A
waiting_for_supplierslist is created to track such consumers and retry linking them when new devices get added.Change above to:
Quoted 39 lines
Quoted 5 lines
If a supplier device has not yet been created when the consumer device attempts to link it, the consumer device is added to the
wait_for_supplierslist. When supplier devices are created, the supplier device link will be added to the relevant consumer devices on thewait_for_supplierslist.I'll take these commit text suggestions if we decide to revert the entire series at the end of this review.
Quoted 29 lines
Quoted 21 lines
Signed-off-by Saravana Kannan <[email protected]>--- drivers/base/core.c | 83 ++++++++++++++++++++++++++++++++++++++++++ include/linux/device.h | 14 +++++++ 2 files changed, 97 insertions(+)drivers/base/core.c +2
-0@@ -44,6 +44,8 @@ early_param("sysfs.deprecated", sysfs_deprecated_setup);#endif /* Device links support. */ +static LIST_HEAD (wait_for_suppliers); +static DEFINE_MUTEX (wfs_lock);#ifdef CONFIG_SRCU static DEFINE_MUTEX (device_links_lock);drivers/base/core.c struct device_link *device_link_add(struct device *consumer, · line 408
Quoted 6 lines
drivers/base/core.c +2
-0@@ -401,6 +403,51 @@ struct device_link *device_link_add(struct device *consumer, }EXPORT_SYMBOL_GPL (device_link_add); +/** +* device_link_wait_for_supplier - Mark device as waiting for supplier *
device_link_wait_for_supplier- Add device towait_for_supplierslistQuoted 7 lines
As a meta-comment, I found this series very hard to understand in the context of reading the new code for the first time. When I read the code again in six months or a year or two years it will not be in near term memory and it will be as if I am reading it for the first time. A lot of my suggestions for changes of names are in that context -- the current names may be fine when one has recently read the code, but not so much when trying to read the whole thing again with a blank mind.
Quoted 7 lines
Thanks for the context.
Quoted 3 lines
The code also inherits a good deal of complexity because it does not stand alone in a nice discrete chunk, but instead delicately weaves into a more complex body of code.
I'll take this as a compliment :)
Please do!
Quoted 15 lines
Quoted 15 lines
When I was trying to understand the code, I wrote a lot of additional comments within my reply email to provide myself context, information about various things, and questions that I needed to answer (or if I could not answer to then ask you). Then I ended up being able to remove many of those notes before sending the reply.
Quoted 6 lines
I intentionally chose "Mark device..." because that's a better description of the semantics of the function instead of trying to describe the implementation. Whether I'm using a linked list or some other data structure should not be the one line documentation of a function. Unless the function is explicitly about operating on that specific data structure.
Quoted 8 lines
Quoted 5 lines
I agree with the intent of trying to describe the semantics of a function, especially at the API level where other systems (or drivers) would be using the function. But for this case the function is at the implementation level and describing explicitly what it is doing makes this much more readable for me.
Are you distinguishing between API level vs implementation level based on the function being "static"/not exported? I believe the earlier
No, being static helps say a function is not API, but an function that is not static may be intended to be used in a limited and constrained manner. I distinguished based on the usage of the function.
Quoted 77 lines
version of this series had this function as an exported API. So maybe that's why I had it as "Mark device".
Quoted 57 lines
I also find "Mark device" to be vague and not descriptive of what the intent is.
Quoted 47 lines
Quoted 36 lines
Quoted 25 lines
drivers/base/core.c +20
-0+* @consumer: Consumer device +* +* Marks the consumer device as waiting for suppliers to become available. The +* consumer device will never be probed until it's unmarked as waiting for +* suppliers. The caller is responsible for adding the link to the supplier +* once the supplier device is present. +* +* This function is NOT meant to be called from the probe function of the +* consumer but rather from code that creates/adds the consumer device. +*/ +static void device_link_wait_for_supplier (struct device *consumer) +{ +mutex_lock (&wfs_lock); +list_add_tail (&consumer->links.needs_suppliers, &wait_for_suppliers); +mutex_unlock (&wfs_lock); +} + +/** +* device_link_check_waiting_consumers - Try to remove from supplier wait list +* drivers/base/core.c struct device_link *device_link_add(struct device *consumer, · line 440
Quoted 10 lines
drivers/base/core.c +10
-0+* Loops through all consumers waiting on suppliers and tries to add all their +* supplier links. If that succeeds, the consumer device is unmarked as waiting +* for suppliers. Otherwise, they are left marked as waiting on suppliers, +* +* The add_links bus callback is expected to return 0 if it has found and added +* all the supplier links for the consumer device. It should return an error if +* it isn't able to do so. +* +* The caller of device_link_wait_for_supplier() is expected to call this once +* it's aware of potential suppliers becoming available. Quoted 6 lines
Change above comment to:
*
device_link_add_supplier_links- add links from consumer devices to
* supplier devices, leaving any consumer
* with inactive suppliers on the
*wait_for_supplierslistI didn't know that the first one line comment could span multiple lines. Good to know.
Quoted 6 lines
Quoted 1 line
* Scan all consumer devices in the devicetree.
This function doesn't have anything to do with devicetree. I've intentionally kept all OF related parts out of the driver/core because I hope that other busses can start using this feature too. So I can't take this bit.
Quoted 8 lines
My comment is left over from when I was taking notes, trying to understand the code.
At the moment, only devicetree is used as a source of the dependency information. The comment would better be re-phrased as:
* Scan all consumer devices in the firmware description of the hardware topology
Ok
Quoted 1 line
I did not ask why this feature is tied to _only_ the platform bus, but will now.
Because devicetree and platform bus the only ones I'm familiar with. If other busses want to add this, I'd be happy to help with code and/or direction/review. But I won't pretend to know anything about ACPI.
Quoted 7 lines
Quoted 2 lines
Sorry, you don't get to ignore other buses because you are not familiar with them.
It's important that I don't design out other buses -- which I don't. But why would you want someone who has no idea of ACPI to write code for it? It's a futile effort that's going to be rejected by people who know ACPI anyway.
Quoted 5 lines
ACPI is not a bus.
Devicetree is not a bus.
A devicetree can contain multiple buses in the topology that is described.
I understand these aren't busses. But most devices from ACPI and DT get put on acpi bus or platform bus? So I was trying to just handle platform bus since that's the majority of the use cases I run into as part of Android. Anyway, see comments further below. I think we are lining up now.
Quoted 28 lines
Quoted 12 lines
Quoted 12 lines
I am not aware of any reason to exclude devices that on other buses and your answer below does not provide a valid technical reason why the new feature is correct when it excludes all other buses.
Quoted 7 lines
Quoted 3 lines
I do not know of any reason that a consumer / supplier relationship can not be between devices on different bus types. Do you know of such a reason?
Yes, it's hypothetically possible. But I haven't seen such a relationship being defined in DT. Nor somewhere else where this might be captured. So, how common/realistic is it?
Quoted 7 lines
Quoted 2 lines
It is entirely legal. I have no idea how common it is but that is not a valid reason to exclude other buses from the feature.
I'm not going to write code for a hypothetical hardware scenario. Find one supported in upstream, show me that it'll benefit from this series and tell me how to interpret the dependency graph and then we'll talk about writing code for that.
You don't get to implement a general feature in a way that only supports a subset of potential devicetree users. Note the word "general". This is not a small isolated feature.
Now, am I being inconsistent if I say that it is ok for the feature to only support devicetree systems, or only support ACPI systems? I'll have to ponder that.
Quoted 7 lines
But I don't think the question of only platform buses or all buses needs to be resolved because I don't think that the
add_linksfunction is a bus specific function. Theadd_linksfunction is specific to devicetree or ACPI.We seem to be talking past each other on this point right now. I don't now how to get our minds to the same place, but let's keep trying.
See my reply further below. That should address some of the concerns as "buses" won't be a concern anymore. Having said that, the main point I was making here is that I can't design for a hypothetical case that has no example with a proper definition of what's the expected behavior.
Quoted 378 lines
Quoted 77 lines
Quoted 69 lines
Quoted 18 lines
Quoted 18 lines
Quoted 11 lines
Quoted 10 lines
For any supplier device that
* is not already linked to the consumer device, add the supplier to the
* consumer device's device links.
*
* If all of a consumer device's suppliers are available then the consumer
* is removed from thewait_for_supplierslist (if previously on the list).
* Otherwise the consumer is added to thewait_for_supplierslist (if not
* already on the list).Quoted 1 line
Honestly, I don't think this is any better than what I already have.
Note that my version of these comments was written while I was reading the code, and did not have any big picture understanding yet. This will likely also be the mind set of most everyone who reads this code in the future, once it is woven into the kernel.
Quoted 41 lines
Quoted 2 lines
If you don't like the change, I can revisit it in a later version of the patch set.
I'll take in all the ones I feel are reasonable or don't feel strongly about. We can revisit the rest later.
Quoted 35 lines
Quoted 15 lines
Quoted 15 lines
* The
add_linksbus callback must return 0 if it has found and added all
* the supplier links for the consumer device. It must return an error if
* it is not able to do so.
*
* The caller ofdevice_link_wait_for_supplier()is expected to call this once
* it is aware of potential suppliers becoming available.drivers/base/core.c struct device_link *device_link_add(struct device *consumer, · line 442
Quoted 2 lines
drivers/base/core.c +2
-0+*/ +static void device_link_check_waiting_consumers (void )Function name is misleading and hides side effects.
Quoted 7 lines
Quoted 4 lines
I have not come up with a name that does not hide side effects, but a better name would be:
device_link_add_supplier_links()I kinda agree that it could afford a better name. The current name is too similar to
device_links_check_suppliers()and I never liked that.Naming new fields or variables related to device links looks pretty challenging to me, because of the desire to be part of device links and not a wart pasted on the side. So I share the pain in trying to find good names.
Quoted 2 lines
Maybe
device_link_add_missing_suppliers()?My first reaction was "yes, that sounds good". But then I stopped and tried to read the name out of context. The name is not adding the missing suppliers, it is saving the information that a supplier is not yet available (eg, is "missing"). I struggled in coming up with
Reading what you say below, and looking at the code again, what I say in that sentence is backwards. It is not adding the missing supplier device links, it is instead adding existing supplier device inks.
Quoted 3 lines
Quoted 1 line
the name that I suggested. We can keep thinking.
No, this function _IS_ about adding links to suppliers. These
Quoted 7 lines
You are mis-reading what I wrote. I said the function "is not adding the missing suppliers". You are converting that to "is not adding links to the missing suppliers".
My suggested name was hinting "
add_supplier_links", which is what you say it does below. The name you suggest is hinting "add_missing_suppliers". Do you see the difference?Quoted 7 lines
Yeah, which is why I said earlier that I didn't want to repeat "links" twice in a function name. As in
device_links_add_missing_supplier_links()has too many "links". In the context ofdevice_links_, "add missing suppliers" means "add missing supplier links". Anyway, I think we can come back to figuring out a good name once we agree on the more important discussions further below.Yes, later is fine. This is a detail.
Quoted 291 lines
Quoted 274 lines
Quoted 20 lines
consumers were "saved" as "not yet having the supplier" earlier by
device_link_wait_for_supplier(). This function doesn't do that. This function is just trying to see if those missing suppliers are present now and if so adding a link to them from the "saved" consumers. I thinkdevice_link_add_missing_suppliers()is actually a pretty good name. Let me know what you think now.Quoted 12 lines
Quoted 4 lines
I don't think we need "links" repeated twice in the function name.
Yeah, I didn't like that either.
Quoted 3 lines
With this suggestion, what side effect is hidden in your opinion? That the fully linked consumer is removed from the "waiting for suppliers" list?
Quoted 6 lines
Quoted 2 lines
The side effect is that the function does not merely do a check. It also adds missing suppliers to a list.
No, it doesn't do that. I can't keep a list of things that aren't allocated yet :). In the whole patch series, we only keep a list of things (consumers) that are waiting on other things (missing suppliers).
OK, as I noted above, I stated that backwards. It is adding links for existing suppliers, not for the missing suppliers.
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
Maybe
device_link_try_removing_from_wfs()?I like that, other than the fact that it still does not provide a clue that the function is potentially adding suppliers to a list.
It doesn't. How would you add a supplier device to a list if the device itself isn't there? :)
Again, that should be existing suppliers, as you noted. But the point stands that the function is potentially adding links.
Quoted 224 lines
Quoted 127 lines
I think
part of the challenge is that the function does two things: (1) a check, and (2) potentially adding missing suppliers to a list. Maybe a simple one line comment at the call site, something like:/* adds missing suppliers to wfs */Quoted 15 lines
I'll wait for us to agree on a better name here before I change this.
Quoted 11 lines
drivers/base/core.c struct device_link *device_link_add(struct device *consumer, · line 450
Quoted 8 lines
drivers/base/core.c +8
-0+{ +struct device *dev, *tmp; + +mutex_lock (&wfs_lock); +list_for_each_entry_safe (dev, tmp, &wait_for_suppliers, + links.needs_suppliers) +if (!dev->bus->add_links (dev)) +list_del_init (&dev->links.needs_suppliers);Empties
dev->links.needs_suppliers, but does not remove dev fromwait_for_supplierslist. Where does that happen?Quoted 3 lines
I'll chalk this up to you having a long day or forgetting your coffee :)
list_del_init()does both of those things becauseneeds_suppliersis the node andwait_for_suppliersis the list.Yes, brain mis-fire on my part. I'll have to go back and look at the list related code again.
Quoted 21 lines
Quoted 21 lines
Quoted 9 lines
drivers/base/core.c +3
-0+mutex_unlock (&wfs_lock); +} +static void
device_link_free(struct device_link *link){ while (refcount_dec_not_one(&link->rpm_active))drivers/base/core.c int device_links_check_suppliers(struct device *dev) · line 590
Quoted 9 lines
drivers/base/core.c +6
-0@@ -535,6 +582,19 @@ int device_links_check_suppliers(struct device *dev)struct device_link *link;int ret =0 ; +/* +* If a device is waiting for one or more suppliers (in +* wait_for_suppliers list), it is not ready to probe yet. So just +* return -EPROBE_DEFER without having to check the links with existing +* suppliers. +*/ Change comment to:
Quoted 6 lines
Quoted 4 lines
/*
* Device waiting for supplier to become available is not allowed
* to probe
*/Po-tay-to. Po-tah-to? I think my comment is just as good.
If just as good and shorter, then better.
Also the original says "it is not ready to probe". That is not correct. It is ready to probe, it is just that the probe attempt will return -EPROBE_DEFER. Nit picky on my part, but tiny things like that mean I have to think harder. I have to think "why is it not ready to probe?". Maybe my version should have instead been something like:
* Device waiting for supplier to become available will return
* -EPROBE_DEFER if probed. Avoid the unneeded processing.Quoted 56 lines
Quoted 10 lines
drivers/base/core.c int device_links_check_suppliers(struct device *dev) · line 598
Quoted 9 lines
drivers/base/core.c +7
-0+mutex_lock (&wfs_lock); +if (!list_empty (&dev->links.needs_suppliers)) { +mutex_unlock (&wfs_lock); +return -EPROBE_DEFER ; + } +mutex_unlock (&wfs_lock); +device_links_write_lock ();Quoted 2 lines
Update
Documentation/driver-api/device_link.rstto reflect the check of&dev->links.needs_suppliersindevice_links_check_suppliers().Thanks! Will do.
Quoted 41 lines
Quoted 40 lines
drivers/base/core.c +18
-0list_for_each_entry (link, &dev->links.suppliers, c_node) {@@ -812,6 +872,10 @@ static void device_links_purge(struct device *dev) {struct device_link *link, *ln; +mutex_lock (&wfs_lock); +list_del (&dev->links.needs_suppliers); +mutex_unlock (&wfs_lock); +/* * Delete all of the remaining links from this device to any other * devices (either consumers or suppliers). @@ -1673,6 +1737,7 @@ void device_initialize(struct device *dev)#endif INIT_LIST_HEAD (&dev->links.consumers);INIT_LIST_HEAD (&dev->links.suppliers); +INIT_LIST_HEAD (&dev->links.needs_suppliers); dev->links.status =DL_DEV_NO_DRIVER ; }EXPORT_SYMBOL_GPL (device_initialize);@@ -2108,6 +2173,24 @@ int device_add(struct device *dev) BUS_NOTIFY_ADD_DEVICE, dev);kobject_uevent (&dev->kobj,KOBJ_ADD ); + +/* +* Check if any of the other devices (consumers) have been waiting for +* this device (supplier) to be added so that they can create a device +* link to it. +* +* This needs to happen after device_pm_add() because device_link_add() +* requires the supplier be registered before it's called. +* +* But this also needs to happe before bus_probe_device() to make sure +* waiting consumers can link to it before the driver is bound to the +* device and the driver sync_state callback is called for this device. +*/ Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
/*
* Add links to dev from any dependent consumer that has dev on it's
* list of needed suppliersThere is no list of needed suppliers.
"the other devices (consumers) have been waiting for this device (supplier)". Isn't that a list of needed suppliers?
No, that's a list of consumers that
needs_suppliers.Quoted 75 lines
Quoted 9 lines
Quoted 7 lines
(links.needs_suppliers).
Device_pm_add()
* must have previously registered dev to allow the links to be added. *
* The consumer links must be created before dev is probed because the
*sync_statecallback for dev will use the consumer links. */I think what I wrote is just as clear.
The original comment is vague. It does not explain why consumer links must be created before the probe. I had to go off and read other code to determine why that is true.
And again, brevity is better if otherwise just as clear.
Quoted 26 lines
Quoted 26 lines
Quoted 25 lines
drivers/base/core.c +5
-0+device_link_check_waiting_consumers (); + +if (dev->bus && dev->bus->add_links && dev->bus->add_links (dev)) +device_link_wait_for_supplier (dev); +bus_probe_device (dev);if (parent)klist_add_tail(&dev->p->knode_parent,include/linux/device.h +6
-0@@ -78,6 +78,17 @@ extern void bus_remove_file(struct bus_type *, struct bus_attribute *);* -EPROBE_DEFER it will queue the device for deferred probing. * @uevent: Called when a device is added, removed, or a few other things * that generate uevents to add the environment variables. +* @add_links: Called, perhaps multiple times per device, after a device is +* added to this bus. The function is expected to create device +* links to all the suppliers of the input device that are +* available at the time this function is called. As in, the +* function should NOT stop at the first failed device link if +* other unlinked supplier devices are present in the system. Quoted 4 lines
Quoted 1 line
* @add_links: Called after a device is added to this bus.
Why are you removing the "perhaps multiple times" part? that's true and that's how some of the other ops are documented.
I didn't remove it. I rephrased it with a little bit more explanation as "If some suppliers are not yet available, this function will be called again when the suppliers become available." (below).
Quoted 9 lines
Quoted 9 lines
The function is
* expected to create device links to all the suppliers of the * device that are available at the time this function is called. * The function must NOT stop at the first failed device link if * other unlinked supplier devices are present in the system. * If some suppliers are not yet available, this function will be * called again when the suppliers become available.Quoted 6 lines
Quoted 1 line
but
add_links()not needed, so moving this comment toof_link_to_suppliers()Sorry, I'm not sure I understand. Can you please explain what you are trying to say?
of_link_to_suppliers()is just one implementation ofadd_links(). The comment above is try for any bus trying to implementadd_links().This is conflating bus with the source of the firmware description of the hardware topology. For drivers that use various APIs to access firmware description of topology that may be either devicetree or ACPI the access is done via
fwnode_operations, based on struct device.fwnode (if I recall properly).Quoted 6 lines
I failed to completely address why
add_links()is not needed. The answer is that there should be a single function called for all buses. Then the proper firmware data source would be accessed via a structfwnode_operations.I think I left this out because I had not yet asked why this feature is tied only to the platform bus. Which I asked earlier in this reply.
Thanks for the pointer about fwnode and
fwnode_operations. I wasn't aware of those. I see where you are going with this. I see a couple of problems with this approach though:Quoted 5 lines
1. How you interpret the properties of a fwnode is specific to the fw type. The clocks DT property isn't going to have the same definition in ACPI or some other firmware. Heck, I don't know if ACPI even has a clocks like property. So have one function to parse all the FW types doesn't make a lot of sense.
Quoted 5 lines
The functions in
fwnode_operationsare specific to the proper firmware. So there is a set of functions in a structfwnode_operationsfor devicetree that only know about devicetree. And there is a different variable of typefwnode_operationsthat is initialized with ACPI specific functions.Yes, I understand how ops work :) So I have one ops (fwnode ops) to call that will read a property from DT or ACPI depending on where that specific device's firmware is from. But that's not my point here.
My point is that clock bindings in DT are under a "clocks" property that lists references (phandles) to the supplier. But in ACPI, the property might be called "clk" and could list references to actual clock IDs. So, you can't have one piece of code that works for all firmware even if I have one ops that can read properties from any firmware.
Quoted 7 lines
Quoted 3 lines
I'll still have to know what type the underlying firmware is before I try to interpret the properties. So having one function that parses DT and ACPI and whatever else would be a terrible and unnecessary design.
You have already implemented the devicetree function, which is
of_link_to_suppliers(). The devicetreefwnode_operationswould have a pointer toof_link_to_suppliers().
Ah, I didn't realize you were asking me to add to the fwnode ops. I thought you wanted me to handle this at the driver core level by moving of_link_to_suppliers to driver core and replacing of_* APIs with fwnode ops. And that seemed like a terrible idea. Glad you weren't suggesting that.
I'm definitely open to adding a add_links to fwnode ops, but I'm not sure if fwnode ops changes are frowned upon or not. I'll still need the device specific edit_links() but that's a separate issue.
Quoted 3 lines
If ACPI support is added, there would be an analogous ACPI aware function that would essentially do the same thing that
of_link_to_suppliers()does. This would be in the ACPI version offwnode_operations.
Agreed. As long as you don't ask me to implement the ACPI ops :)
Quoted 2 lines
There would not be a single function that is both devicetree aware and ACPI aware.
Great.
Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Quoted 5 lines
2. If this common code is implemented as part of driver/base/, then at a minimum, I'll have to check if a fwnode is a DT node before I start interpreting the properties of a device's fwnode. But that means I'll have to include
linux/of.hto useis_of_node(). I don't like having driver/base code depend on OF or platform or ACPI headers.You just use the function in the device's
fwnode_operations(I think, I would have to go look at the precise way the code works because it has been quite a while since I've looked at it).
Quoted 6 lines
Quoted 2 lines
Because you missed my point in (1) you are missing my point in (2). I'll wait for your updated reply.
We are still talking at cross purposes. If my reply to (1) does not change that, I'll have to go dig into how the fwnode framework figures out which set of
fwnode_operationsto use for each device.
I think we are lining up better now. But still have some more to go :)
Having said that, I'd still like to meet your tomorrow if that's possible (see Greg's email).
-Saravana
Re: [PATCH v7 1/7] driver core: Add support for linking devices during device addition
In reply to Saravana KannanQuoted 5 lines · Saravana Kannan
Quoted 4 lines · Frank Rowand
Quoted 2 lines · Saravana Kannan
Quoted 1 line · Frank Rowand
< snip >
Quoted 11 lines
Quoted 7 lines
Quoted 3 lines
3. The supplier info doesn't always need to come from a firmware. So I don't want to limit it to that?
If you can find another source of topology info, then I would expect that another set of
fwnode_operationsfunctions would be created for the info source.The other source could just be C files in the kernel. Using fwnodes for that would be hacky. But let's sort (1) and (2) out first.
< snip >
Just a piece of trivia. I got curious enough about this to search. There is a third type of fwnode, software nodes.
See commit 59abd83672f70cac4b6bf9b237506c5bc6837606 for a description.
-Frank
PATCH v7 2/7 driver core: Add edit_links() callback for drivers
In reply to Saravana KannanThe driver core/bus adding supplier-consumer dependencies by default enables functional dependencies to be tracked correctly even when the consumer devices haven't had their drivers registered or loaded (if they are modules).
However, when the bus incorrectly adds dependencies that it shouldn't have added, the devices might never probe.
For example, if device-C is a consumer of device-S and they have phandles to each other in DT, the following could happen:
1. Device-S get added first.
2. The bus add_links() callback will (incorrectly) try to link it as a consumer of device-C.
3. Since device-C isn't present, device-S will be put in "waiting-for-supplier" list.
4. Device-C gets added next. 5. All devices in "waiting-for-supplier" list are retried for linking. 6. Device-S gets linked as consumer to Device-C. 7. The bus add_links() callback will (correctly) try to link it as
a consumer of device-S.
8. This isn't allowed because it would create a cyclic device links.
Neither devices will get probed since the supplier is marked as dependent on the consumer. And the consumer will never probe because the consumer can't get resources from the supplier.
Without this patch, things stay in this broken state. However, with this patch, the execution will continue like this:
9. Device-C's driver is loaded.
10. Device-C's driver removes Device-S as a consumer of Device-C.
11. Device-C's driver adds Device-C as a consumer of Device-S.
12. Device-S probes.
14. Device-C probes.
--- drivers/base/core.c | 24 ++++++++++++++++++++++-- drivers/base/dd.c | 29 +++++++++++++++++++++++++++++ include/linux/device.h | 18 ++++++++++++++++++ 3 files changed, 69 insertions(+), 2 deletions(-)
drivers/base/core.c +22 -2
@@ -422,6 +422,19 @@ static void device_link_wait_for_supplier(struct device *consumer)mutex_unlock (&wfs_lock); } +/** +* device_link_remove_from_wfs - Unmark device as waiting for supplier +* @consumer: Consumer device +* +* Unmark the consumer device as waiting for suppliers to become available. +*/ +void device_link_remove_from_wfs (struct device *consumer) +{ +mutex_lock (&wfs_lock); +list_del_init (&consumer->links.needs_suppliers); +mutex_unlock (&wfs_lock); +} +/** * device_link_check_waiting_consumers - Try to unmark waiting consumers *
@@ -439,12 +452,19 @@ static void device_link_wait_for_supplier(struct device *consumer)static void device_link_check_waiting_consumers (void ) {struct device *dev, *tmp; +int ret;mutex_lock (&wfs_lock);list_for_each_entry_safe (dev, tmp, &wait_for_suppliers, - links.needs_suppliers) -if (!dev->bus->add_links (dev)) + links.needs_suppliers) { + ret =0 ; +if (dev->has_edit_links) + ret =driver_edit_links (dev); +else if (dev->bus->add_links) + ret = dev->bus->add_links (dev); +if (!ret)list_del_init (&dev->links.needs_suppliers); + }mutex_unlock (&wfs_lock); }
drivers/base/dd.c +29 -0
@@ -698,6 +698,12 @@ int driver_probe_device(struct device_driver *drv, struct device *dev)pr_debug ("bus: '%s': %s: matched device %s with driver %s\n" , drv->bus->name, __func__,dev_name (dev), drv->name); +if (drv->edit_links) { +if (drv->edit_links (dev)) + dev->has_edit_links = true; +else +device_link_remove_from_wfs (dev); + }pm_runtime_get_suppliers (dev);if (dev->parent)pm_runtime_get_sync (dev->parent);
@@ -786,6 +792,29 @@ struct device_attach_data {bool have_async; }; +static int __driver_edit_links (struct device_driver *drv,void *data) +{ +struct device *dev = data; + +if (!drv->edit_links) +return 0 ; + +if (driver_match_device (drv, dev) <=0 ) +return 0 ; + +return drv->edit_links (dev); +} + +int driver_edit_links (struct device *dev) +{ +int ret; + +device_lock (dev); + ret =bus_for_each_drv (dev->bus,NULL , dev, __driver_edit_links); +device_unlock (dev); +return ret; +} +static int __device_attach_driver (struct device_driver *drv,void *_data) {struct device_attach_data *data = _data;
include/linux/device.h +18 -0
@@ -263,6 +263,20 @@ enum probe_type {* @probe_type: Type of the probe (synchronous or asynchronous) to use. * @of_match_table: The open firmware table. * @acpi_match_table: The ACPI match table. +* @edit_links: Called to allow a matched driver to edit the device links the +* bus might have added incorrectly. This will be useful to handle +* cases where the bus incorrectly adds functional dependencies +* that aren't true or tries to create cyclic dependencies. But +* doesn't correctly handle functional dependencies that are +* missed by the bus as the supplier's sync_state might get to +* execute before the driver for a missing consumer is loaded and +* gets to edit the device links for the consumer. +* +* This function might be called multiple times after a new device +* is added. The function is expected to create all the device +* links for the new device and return 0 if it was completed +* successfully or return an error if it needs to be reattempted +* in the future. * @probe: Called to query the existence of a specific device, * whether this driver can work with it, and bind the driver * to a specific device.
@@ -302,6 +316,7 @@ struct device_driver {const struct of_device_id *of_match_table;const struct acpi_device_id *acpi_match_table; +int (*edit_links)(struct device *dev);int (*probe) (struct device *dev);int (*remove) (struct device *dev);void (*shutdown) (struct device *dev);
@@ -1078,6 +1093,7 @@ struct device { bool offline_disabled:1; bool offline:1; bool of_node_reused:1; + bool has_edit_links:1;#if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_DEVICE ) || \ defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU ) || \ defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU_ALL )
@@ -1329,6 +1345,7 @@ extern int __must_check device_attach(struct device *dev);extern int __must_check driver_attach (struct device_driver *drv);extern void device_initial_probe (struct device *dev);extern int __must_check device_reprobe (struct device *dev); +extern int driver_edit_links (struct device *dev);extern bool device_is_bound (struct device *dev);
@@ -1419,6 +1436,7 @@ struct device_link *device_link_add(struct device *consumer, struct device *supplier, u32 flags);void device_link_del (struct device_link *link);void device_link_remove (void *consumer,struct device *supplier); +void device_link_remove_from_wfs (struct device *consumer);#ifndef dev_fmt#define dev_fmt (fmt) fmt
Signature · 1 line
2.22.0.709.g102302147b-goog
Re: [PATCH v7 2/7] driver core: Add edit_links() callback for drivers
Quoted 9 lines
Date: Tue, 23 Jul 2019 17:10:55 -0700
Subject: [PATCH v7 2/7] driver core: Addedit_links()callback for drivers
From: Saravana Kannan <[email protected]>The driver core/bus adding supplier-consumer dependencies by default
enables functional dependencies to be tracked correctly even when the consumer devices haven't had their drivers registered or loaded (if they are modules).
enables functional dependencies to be tracked correctly before the consumer device drivers are registered or loaded (if they are modules).
Quoted 2 lines
However, when the bus incorrectly adds dependencies that it shouldn't
^^^ driver core/bus
Quoted 1 line
have added, the devices might never probe.
Explain what causes a dependency to be incorrectly added.
Is this a bug in the dependency detection code?
Are there cases where the dependency detection code can not reliably determine whether there truly is a dependency?
Quoted 16 lines
For example, if device-C is a consumer of device-S and they have phandles to each other in DT, the following could happen:
1. Device-S get added first.
2. The busadd_links()callback will (incorrectly) try to link it as a consumer of device-C.
3. Since device-C isn't present, device-S will be put in "waiting-for-supplier" list.4. Device-C gets added next. 5. All devices in "waiting-for-supplier" list are retried for linking. 6. Device-S gets linked as consumer to Device-C. 7. The bus add_links() callback will (correctly) try to link it asa consumer of device-S.
8. This isn't allowed because it would create a cyclic device links.
Quoted 8 lines
Neither devices will get probed since the supplier is marked as dependent on the consumer. And the consumer will never probe because the consumer can't get resources from the supplier.
Without this patch, things stay in this broken state. However, with this patch, the execution will continue like this:
9. Device-C's driver is loaded.
Change comment to:
For example, if device-C is a consumer of device-S and they have phandles referencing each other in the devicetree, the following could happen:
1. Device-S is added first.
- The bus add_links() callback will (incorrectly) link device-S as a consumer of device-C, and device-S will be put in the "wait_for_suppliers" list.
2. Device-C is added next.
- All devices in the "wait_for_suppliers" list are retried for linking.
- Device-S remains linked as a consumer to device-C.
- The bus add_links() callback will (correctly) try to link device-C as a consumer of device-S.
- The link attempt will fail because it would create a cyclic device link, and device-C will be put in the "wait_for_suppliers" list.
Device-S will not be probed because it is in the "wait_for_suppliers" list. Device-C will not be probed because it is in the "wait_for_suppliers" list.
Quoted 5 lines
Without this patch, things stay in this broken state. However, with this patch, the execution will continue like this:
9. Device-C's driver is loaded.
What is "loaded"? Does that mean the device-C probe succeeds?
What causes device-C to be probed? The normal processing of -EPROBE_DEFER devices?
Quoted 17 lines
10. Device-C's driver removes Device-S as a consumer of Device-C.
11. Device-C's driver adds Device-C as a consumer of Device-S.
12. Device-S probes.
14. Device-C probes.Signed-off-by Saravana Kannan <[email protected]>--- drivers/base/core.c | 24 ++++++++++++++++++++++-- drivers/base/dd.c | 29 +++++++++++++++++++++++++++++ include/linux/device.h | 18 ++++++++++++++++++ 3 files changed, 69 insertions(+), 2 deletions(-)drivers/base/core.c +0
-0
drivers/base/core.c static void device_link_wait_for_supplier(struct device *consumer) · line 431
Quoted 10 lines
drivers/base/core.c +7
-0@@ -422,6 +422,19 @@ static void device_link_wait_for_supplier(struct device *consumer)mutex_unlock (&wfs_lock); } +/** +* device_link_remove_from_wfs - Unmark device as waiting for supplier +* @consumer: Consumer device +* +* Unmark the consumer device as waiting for suppliers to become available. +*/ +void device_link_remove_from_wfs (struct device *consumer)
Misleading function name.
Incorrect description.
Does not remove consumer from list wait_for_suppliers.
At best, consumer might eventually get removed from list wait_for_suppliers if device_link_check_waiting_consumers() is called again.
Quoted 36 lines
drivers/base/core.c +15
-2+{ +mutex_lock (&wfs_lock); +list_del_init (&consumer->links.needs_suppliers); +mutex_unlock (&wfs_lock); +} +/** * device_link_check_waiting_consumers - Try to unmark waiting consumers * @@ -439,12 +452,19 @@ static void device_link_wait_for_supplier(struct device *consumer)static void device_link_check_waiting_consumers (void ) {struct device *dev, *tmp; +int ret;mutex_lock (&wfs_lock);list_for_each_entry_safe (dev, tmp, &wait_for_suppliers, - links.needs_suppliers) -if (!dev->bus->add_links (dev)) + links.needs_suppliers) { + ret =0 ; +if (dev->has_edit_links) + ret =driver_edit_links (dev); +else if (dev->bus->add_links) + ret = dev->bus->add_links (dev); +if (!ret)list_del_init (&dev->links.needs_suppliers); + }mutex_unlock (&wfs_lock); }drivers/base/dd.c +0
-0
drivers/base/dd.c int driver_probe_device(struct device_driver *drv, struct device *dev) · line 706
Quoted 9 lines
drivers/base/dd.c +6
-0@@ -698,6 +698,12 @@ int driver_probe_device(struct device_driver *drv, struct device *dev)pr_debug ("bus: '%s': %s: matched device %s with driver %s\n" , drv->bus->name, __func__,dev_name (dev), drv->name); +if (drv->edit_links) { +if (drv->edit_links (dev)) + dev->has_edit_links = true; +else +device_link_remove_from_wfs (dev); + }
For the purposes of the following paragraphs, I refer to dev as "dev_1" to distinguish it from a new dev that will be encountered later. The following paragraphs assume dev_1 has a supplier dependency for a supplier that has not probed yet.
Q. Why the extra level of indirection?
A. really_probe() does not set dev->driver before returning if
device_links_check_suppliers() returned -EPROBE_DEFER. Thus device_link_check_waiting_consumers() can not directly check "if (dev_1->driver->edit_links)".
The added driver_probe_device() code is setting dev_1->has_edit_links in the probe path, then device_link_check_waiting_consumers() will use the value of dev_1->has_edit_links instead of directly checking
"if (dev_1->driver->edit_links)".
If really_probe() was modified to set dev->driver in this case then the need for dev->has_edit_links is removed and driver_edit_links() is not needed, since dev->driver would be available. Removing driver_edit_links() simplifies the code.
device_add() calls dev_1->bus->add_links(dev_1), thus dev_1 will have the supplier links set (for any suppliers not currently available) and be on list wait_for_suppliers.
Then device_add() calls bus_probe_device(), leading to calling driver_probe_device(). The above code fragment either sets dev_1->has_edit_links or removes the needs_suppliers links from dev_1. dev_1 remains on list wait_for_suppliers.
If (drv->edit_links(dev_1) returns 0 then device_link_remove_from_wfs() removes the supplier links. Shouldn't device_link_remove_from_wfs() also remove the device from the list wait_for_suppliers?
The next time a device is added, device_link_check_waiting_consumers() will be called and dev_1 will be on list wait_for_suppliers, thus device_link_check_waiting_consumers() will find dev_1->has_edit_links true and thus call driver_edit_links() instead of calling dev->bus->add_links().
The comment in device.h, later in this patch, says that drv->edit_links() is responsible for editing the device links for dev. The comment provides no guidance on how drv->edit_links() is supposed to determine what edits to perform. No example drv->edit_links() function is provided in this patch series. dev_1->bus->add_links(dev_1) may have added one or more suppliers to its needs_suppliers link. drv->edit_links() needs to be able to handle all possible variants of what suppliers are on the needs_suppliers link.
Quoted 38 lines
drivers/base/dd.c +23
-0pm_runtime_get_suppliers (dev);if (dev->parent)pm_runtime_get_sync (dev->parent);@@ -786,6 +792,29 @@ struct device_attach_data {bool have_async; }; +static int __driver_edit_links (struct device_driver *drv,void *data) +{ +struct device *dev = data; + +if (!drv->edit_links) +return 0 ; + +if (driver_match_device (drv, dev) <=0 ) +return 0 ; + +return drv->edit_links (dev); +} + +int driver_edit_links (struct device *dev) +{ +int ret; + +device_lock (dev); + ret =bus_for_each_drv (dev->bus,NULL , dev, __driver_edit_links); +device_unlock (dev); +return ret; +} +static int __device_attach_driver (struct device_driver *drv,void *_data) {struct device_attach_data *data = _data;include/linux/device.h +0
-0
include/linux/device.h enum probe_type { · line 266
Quoted 4 lines
include/linux/device.h +1
-0@@ -263,6 +263,20 @@ enum probe_type {* @probe_type: Type of the probe (synchronous or asynchronous) to use. * @of_match_table: The open firmware table. * @acpi_match_table: The ACPI match table. +* @edit_links: Called to allow a matched driver to edit the device links the
Where is the value of field edit_links set?
Is it only in an out of tree driver? If so, I would like to see an example implementation of the edit_links() function.
Quoted 25 lines
include/linux/device.h +14
-0+* bus might have added incorrectly. This will be useful to handle +* cases where the bus incorrectly adds functional dependencies +* that aren't true or tries to create cyclic dependencies. But +* doesn't correctly handle functional dependencies that are +* missed by the bus as the supplier's sync_state might get to +* execute before the driver for a missing consumer is loaded and +* gets to edit the device links for the consumer. +* +* This function might be called multiple times after a new device +* is added. The function is expected to create all the device +* links for the new device and return 0 if it was completed +* successfully or return an error if it needs to be reattempted +* in the future. * @probe: Called to query the existence of a specific device, * whether this driver can work with it, and bind the driver * to a specific device. @@ -302,6 +316,7 @@ struct device_driver {const struct of_device_id *of_match_table;const struct acpi_device_id *acpi_match_table; +int (*edit_links)(struct device *dev);int (*probe) (struct device *dev);int (*remove) (struct device *dev);void (*shutdown) (struct device *dev);
include/linux/device.h struct device { · line 1096
Quoted 4 lines
include/linux/device.h +1
-0@@ -1078,6 +1093,7 @@ struct device { bool offline_disabled:1; bool offline:1; bool of_node_reused:1; + bool has_edit_links:1;
Add has_edit_links to the struct's kernel_doc
Quoted 23 lines
include/linux/device.h +2
-0#if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_DEVICE ) || \ defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU ) || \ defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU_ALL )@@ -1329,6 +1345,7 @@ extern int __must_check device_attach(struct device *dev);extern int __must_check driver_attach (struct device_driver *drv);extern void device_initial_probe (struct device *dev);extern int __must_check device_reprobe (struct device *dev); +extern int driver_edit_links (struct device *dev);extern bool device_is_bound (struct device *dev);@@ -1419,6 +1436,7 @@ struct device_link *device_link_add(struct device *consumer, struct device *supplier, u32 flags);void device_link_del (struct device_link *link);void device_link_remove (void *consumer,struct device *supplier); +void device_link_remove_from_wfs (struct device *consumer);#ifndef dev_fmt#define dev_fmt (fmt) fmtSignature · 1 line
2.22.0.709.g102302147b-goog
Re: [PATCH v7 2/7] driver core: Add edit_links() callback for drivers
Quoted 15 lines · Frank Rowand
Quoted 10 lines
Date: Tue, 23 Jul 2019 17:10:55 -0700
Subject: [PATCH v7 2/7] driver core: Addedit_links()callback for drivers
From: Saravana Kannan <[email protected]>The driver core/bus adding supplier-consumer dependencies by default
enables functional dependencies to be tracked correctly even when the consumer devices haven't had their drivers registered or loaded (if they are modules).
enables functional dependencies to be tracked correctly before the consumer device drivers are registered or loaded (if they are modules).
Quoted 1 line
Quoted 7 lines
Quoted 1 line
However, when the bus incorrectly adds dependencies that it shouldn't
^^^ driver core/bus
Quoted 1 line
have added, the devices might never probe.
Explain what causes a dependency to be incorrectly added.
That depends on the bus specific implementation of add_links()? Not sure I can explain how a future implementation can get it wrong?
Quoted 1 line
Is this a bug in the dependency detection code?
Yes, as in, thinking it's a dependency when it's not.
Quoted 2 lines
Are there cases where the dependency detection code can not reliably determine whether there truly is a dependency?
Correct. One example is the clock controller cyclic dependency that was mentioned in the v1 patch series [1]. Search for "cyclic" if you want to read that part. Can happen between interconnect providers too for example. But I explain it below in the commit text as Device-C and Device-S. Both of those could be different clock providers.
Quoted 30 lines
Quoted 16 lines
For example, if device-C is a consumer of device-S and they have phandles to each other in DT, the following could happen:
1. Device-S get added first.
2. The busadd_links()callback will (incorrectly) try to link it as a consumer of device-C.
3. Since device-C isn't present, device-S will be put in "waiting-for-supplier" list.4. Device-C gets added next. 5. All devices in "waiting-for-supplier" list are retried for linking. 6. Device-S gets linked as consumer to Device-C. 7. The bus add_links() callback will (correctly) try to link it asa consumer of device-S.
8. This isn't allowed because it would create a cyclic device links.Quoted 8 lines
Neither devices will get probed since the supplier is marked as dependent on the consumer. And the consumer will never probe because the consumer can't get resources from the supplier.
Without this patch, things stay in this broken state. However, with this patch, the execution will continue like this:
9. Device-C's driver is loaded.
Change comment to:
For example, if device-C is a consumer of device-S and they have phandles referencing each other in the devicetree, the following could happen:
Quoted 8 lines
1. Device-S is added first.
- The busadd_links()callback will (incorrectly) link device-S as a consumer of device-C, and device-S will be put in the "wait_for_suppliers" list.2. Device-C is added next.
- All devices in the "wait_for_suppliers" list are retried for linking.
- Device-S remains linked as a consumer to device-C.
Device-S gets linked. Not remains linked.
Quoted 6 lines
- The bus
add_links()callback will (correctly) try to link device-C as a consumer of device-S.
- The link attempt will fail because it would create a cyclic device link, and device-C will be put in the "wait_for_suppliers" list.Device-S will not be probed because it is in the "
wait_for_suppliers" list.
Not correct. Device-S will not be probed because it has a device link to Device-C.
Quoted 1 line
Device-C will not be probed because it is in the "
wait_for_suppliers" list.
Correct.
Quoted 8 lines
Quoted 6 lines
Without this patch, things stay in this broken state. However, with this patch, the execution will continue like this:
9. Device-C's driver is loaded.
What is "loaded"? Does that mean the device-C probe succeeds?
No, module loading. I was using a loadable driver as an example. The same thing could happen if the driver is registered at a later init call level too.
Quoted 2 lines
What causes device-C to be probed? The normal processing of -EPROBE_DEFER devices?
It's not probed.
Quoted 20 lines
Quoted 20 lines
10. Device-C's driver removes Device-S as a consumer of Device-C.
11. Device-C's driver adds Device-C as a consumer of Device-S.
12. Device-S probes.
14. Device-C probes.Signed-off-by Saravana Kannan <[email protected]>--- drivers/base/core.c | 24 ++++++++++++++++++++++-- drivers/base/dd.c | 29 +++++++++++++++++++++++++++++ include/linux/device.h | 18 ++++++++++++++++++ 3 files changed, 69 insertions(+), 2 deletions(-)drivers/base/core.c +0
-0@@ -422,6 +422,19 @@ static void device_link_wait_for_supplier(struct device *consumer)mutex_unlock (&wfs_lock); }
Quoted 10 lines
drivers/base/core.c static void device_link_wait_for_supplier(struct device *consumer) · line 431
Quoted 7 lines
drivers/base/core.c +7
-0+/** +* device_link_remove_from_wfs - Unmark device as waiting for supplier +* @consumer: Consumer device +* +* Unmark the consumer device as waiting for suppliers to become available. +*/ +void device_link_remove_from_wfs (struct device *consumer)Misleading function name.
Incorrect description.
See other reply. I think you mixed up a node in a list with a list. The name is very correct and is not misleading.
Quoted 1 line
Does not remove consumer from list
wait_for_suppliers.
It does.
Quoted 66 lines
At best, consumer might eventually get removed from list
wait_for_suppliersifdevice_link_check_waiting_consumers()is called again.Quoted 36 lines
drivers/base/core.c +15
-2+{ +mutex_lock (&wfs_lock); +list_del_init (&consumer->links.needs_suppliers); +mutex_unlock (&wfs_lock); +} +/** * device_link_check_waiting_consumers - Try to unmark waiting consumers * @@ -439,12 +452,19 @@ static void device_link_wait_for_supplier(struct device *consumer)static void device_link_check_waiting_consumers (void ) {struct device *dev, *tmp; +int ret;mutex_lock (&wfs_lock);list_for_each_entry_safe (dev, tmp, &wait_for_suppliers, - links.needs_suppliers) -if (!dev->bus->add_links (dev)) + links.needs_suppliers) { + ret =0 ; +if (dev->has_edit_links) + ret =driver_edit_links (dev); +else if (dev->bus->add_links) + ret = dev->bus->add_links (dev); +if (!ret)list_del_init (&dev->links.needs_suppliers); + }mutex_unlock (&wfs_lock); }drivers/base/dd.c +0
-0drivers/base/dd.c int driver_probe_device(struct device_driver *drv, struct device *dev) · line 706
Quoted 9 lines
drivers/base/dd.c +6
-0@@ -698,6 +698,12 @@ int driver_probe_device(struct device_driver *drv, struct device *dev)pr_debug ("bus: '%s': %s: matched device %s with driver %s\n" , drv->bus->name, __func__,dev_name (dev), drv->name); +if (drv->edit_links) { +if (drv->edit_links (dev)) + dev->has_edit_links = true; +else +device_link_remove_from_wfs (dev); + }For the purposes of the following paragraphs, I refer to dev as "
dev_1" to distinguish it from a new dev that will be encountered later. The following paragraphs assumedev_1has a supplier dependency for a supplier that has not probed yet.Q. Why the extra level of indirection?
A.
really_probe()does not setdev->driverbefore returning if
device_links_check_suppliers()returned -EPROBE_DEFER. Thusdevice_link_check_waiting_consumers()can not directly check "if (dev_1->driver->edit_links)".The added
driver_probe_device()code is settingdev_1->has_edit_linksin the probe path, thendevice_link_check_waiting_consumers()will use the value ofdev_1->has_edit_linksinstead of directly checking
"if (dev_1->driver->edit_links)".
Quoted 5 lines
If
really_probe()was modified to setdev->driverin this case then the need fordev->has_edit_linksis removed anddriver_edit_links()is not needed, sincedev->driverwould be available. Removingdriver_edit_links()simplifies the code.
really_probe() doesn't set device->drv for a good reason. Because the driver could be unregistered/unloaded at anytime before the device and driver are bound to each other.
Quoted 18 lines
device_add()callsdev_1->bus->add_links(dev_1), thusdev_1will have the supplier links set (for any suppliers not currently available) and be on listwait_for_suppliers.Then
device_add()callsbus_probe_device(), leading to callingdriver_probe_device(). The above code fragment either setsdev_1->has_edit_linksor removes theneeds_supplierslinks fromdev_1.dev_1remains on listwait_for_suppliers.If (
drv->edit_links(dev_1)returns 0 thendevice_link_remove_from_wfs()removes the supplier links. Shouldn'tdevice_link_remove_from_wfs()also remove the device from the listwait_for_suppliers?The next time a device is added,
device_link_check_waiting_consumers()will be called anddev_1will be on listwait_for_suppliers, thusdevice_link_check_waiting_consumers()will finddev_1->has_edit_linkstrue and thus calldriver_edit_links()instead of callingdev->bus->add_links().
Quoted 7 lines
The comment in
device.h, later in this patch, says thatdrv->edit_links()is responsible for editing the device links for dev. The comment provides no guidance on howdrv->edit_links()is supposed to determine what edits to perform. No exampledrv->edit_links()function is provided in this patch series.dev_1->bus->add_links(dev_1)may have added one or more suppliers to itsneeds_supplierslink.drv->edit_links()needs to be able to handle all possible variants of what suppliers are on theneeds_supplierslink.
Looks like a significant chunk of this question ties into the assumption that needs_suppliers vs wait_for_suppliers are two different lists. Since I'm clarifying that's not the case, I'll wait to see if you still have any questions and if so wait for your to rewrite this.
Quoted 47 lines
Quoted 40 lines
drivers/base/dd.c +23
-0pm_runtime_get_suppliers (dev);if (dev->parent)pm_runtime_get_sync (dev->parent);@@ -786,6 +792,29 @@ struct device_attach_data {bool have_async; }; +static int __driver_edit_links (struct device_driver *drv,void *data) +{ +struct device *dev = data; + +if (!drv->edit_links) +return 0 ; + +if (driver_match_device (drv, dev) <=0 ) +return 0 ; + +return drv->edit_links (dev); +} + +int driver_edit_links (struct device *dev) +{ +int ret; + +device_lock (dev); + ret =bus_for_each_drv (dev->bus,NULL , dev, __driver_edit_links); +device_unlock (dev); +return ret; +} +static int __device_attach_driver (struct device_driver *drv,void *_data) {struct device_attach_data *data = _data;include/linux/device.h +0
-0include/linux/device.h enum probe_type { · line 266
Quoted 4 lines
include/linux/device.h +1
-0@@ -263,6 +263,20 @@ enum probe_type {* @probe_type: Type of the probe (synchronous or asynchronous) to use. * @of_match_table: The open firmware table. * @acpi_match_table: The ACPI match table. +* @edit_links: Called to allow a matched driver to edit the device links the Where is the value of field
edit_linksset?
Quoted 2 lines
Is it only in an out of tree driver? If so, I would like to see an example implementation of the
edit_links()function.
In the example above, it just needs to delete the Device-S is a consumer of Device-C link and then add a Device-C is a consumer of Device-S link.
I can send an example code in a subsequent reply.
Quoted 20 lines
Quoted 20 lines
include/linux/device.h +13
-0+* bus might have added incorrectly. This will be useful to handle +* cases where the bus incorrectly adds functional dependencies +* that aren't true or tries to create cyclic dependencies. But +* doesn't correctly handle functional dependencies that are +* missed by the bus as the supplier's sync_state might get to +* execute before the driver for a missing consumer is loaded and +* gets to edit the device links for the consumer. +* +* This function might be called multiple times after a new device +* is added. The function is expected to create all the device +* links for the new device and return 0 if it was completed +* successfully or return an error if it needs to be reattempted +* in the future. * @probe: Called to query the existence of a specific device, * whether this driver can work with it, and bind the driver * to a specific device. @@ -302,6 +316,7 @@ struct device_driver {const struct of_device_id *of_match_table;const struct acpi_device_id *acpi_match_table;
Quoted 11 lines
include/linux/device.h struct device { · line 1096
Quoted 9 lines
include/linux/device.h +2
-0+int (*edit_links)(struct device *dev);int (*probe) (struct device *dev);int (*remove) (struct device *dev);void (*shutdown) (struct device *dev);@@ -1078,6 +1093,7 @@ struct device { bool offline_disabled:1; bool offline:1; bool of_node_reused:1; + bool has_edit_links:1;Add
has_edit_linksto the struct'skernel_doc
Already addressed that in a separate patch.
Thanks,
Saravana
[1] https://lore.kernel.org/lkml/CAGETcx-KwwjNgAy7BLv4+1=5N_s-UdmfSnTtHP8V5gc7t48W=Q@mail.gmail.com/
PATCH v7 3/7 of/platform: Add functional dependency link from DT bindings
In reply to Saravana KannanAdd device-links after the devices are created (but before they are probed) by looking at common DT bindings like clocks and interconnects.
Automatically adding device-links for functional dependencies at the framework level provides the following benefits:
- Optimizes device probe order and avoids the useless work of attempting probes of devices that will not probe successfully (because their suppliers aren't present or haven't probed yet).
For example, in a commonly available mobile SoC, registering just one consumer device's driver at an initcall level earlier than the supplier device's driver causes 11 failed probe attempts before the consumer device probes successfully. This was with a kernel with all the drivers statically compiled in. This problem gets a lot worse if all the drivers are loaded as modules without direct symbol dependencies.
- Supplier devices like clock providers, interconnect providers, etc need to keep the resources they provide active and at a particular state(s) during boot up even if their current set of consumers don't request the resource to be active. This is because the rest of the consumers might not have probed yet and turning off the resource before all the consumers have probed could lead to a hang or undesired user experience.
Some frameworks (Eg: regulator) handle this today by turning off "unused" resources at late_initcall_sync and hoping all the devices have probed by then. This is not a valid assumption for systems with loadable modules. Other frameworks (Eg: clock) just don't handle this due to the lack of a clear signal for when they can turn off resources. This leads to downstream hacks to handle cases like this that can easily be solved in the upstream kernel.
By linking devices before they are probed, we give suppliers a clear count of the number of dependent consumers. Once all of the consumers are active, the suppliers can turn off the unused resources without making assumptions about the number of consumers.
By default we just add device-links to track "driver presence" (probe succeeded) of the supplier device. If any other functionality provided by device-links are needed, it is left to the consumer/supplier devices to change the link when they probe.
--- .../admin-guide/kernel-parameters.txt | 5 + drivers/of/platform.c | 165 ++++++++++++++++++ 2 files changed, 170 insertions(+)
Documentation/admin-guide/kernel-parameters.txt +5 -0
@@ -3170,6 +3170,11 @@ This can be set from sysctl after boot. See Documentation/admin-guide/sysctl/vm.rst for details. + of_devlink [KNL] Make device links from common DT bindings. Useful + for optimizing probe order and making sure resources + aren't turned off before the consumer devices have + probed. + ohci1394_dma=early [HW] enable debugging via the ohci1394 driver. See Documentation/debugging-via-ohci1394.txt for more info.
drivers/of/platform.c +165 -0
177 lines not on this page · raw message
Signature · 1 line
2.22.0.709.g102302147b-goog
Re: [PATCH v7 3/7] of/platform: Add functional dependency link from DT bindings
Quoted 45 lines · Saravana Kannan
Add device-links after the devices are created (but before they are probed) by looking at common DT bindings like clocks and interconnects.
Automatically adding device-links for functional dependencies at the framework level provides the following benefits:
- Optimizes device probe order and avoids the useless work of attempting probes of devices that will not probe successfully (because their suppliers aren't present or haven't probed yet).
For example, in a commonly available mobile SoC, registering just one consumer device's driver at an initcall level earlier than the supplier device's driver causes 11 failed probe attempts before the consumer device probes successfully. This was with a kernel with all the drivers statically compiled in. This problem gets a lot worse if all the drivers are loaded as modules without direct symbol dependencies.
- Supplier devices like clock providers, interconnect providers, etc need to keep the resources they provide active and at a particular state(s) during boot up even if their current set of consumers don't request the resource to be active. This is because the rest of the consumers might not have probed yet and turning off the resource before all the consumers have probed could lead to a hang or undesired user experience.
Some frameworks (Eg: regulator) handle this today by turning off "unused" resources at
late_initcall_syncand hoping all the devices have probed by then. This is not a valid assumption for systems with loadable modules. Other frameworks (Eg: clock) just don't handle this due to the lack of a clear signal for when they can turn off resources. This leads to downstream hacks to handle cases like this that can easily be solved in the upstream kernel.By linking devices before they are probed, we give suppliers a clear count of the number of dependent consumers. Once all of the consumers are active, the suppliers can turn off the unused resources without making assumptions about the number of consumers.
By default we just add device-links to track "driver presence" (probe succeeded) of the supplier device. If any other functionality provided by device-links are needed, it is left to the consumer/supplier devices to change the link when they probe.
Quoted 6 lines
Signed-off-by Saravana Kannan <[email protected]>--- .../admin-guide/kernel-parameters.txt | 5 + drivers/of/platform.c | 165 ++++++++++++++++++ 2 files changed, 170 insertions(+)
Documentation/admin-guide/kernel-paramers.rst:
After line 129, add:
OF Devicetree is enabled
Documentation/admin-guide/kernel-parameters.txt line 3176
Quoted 12 lines
Documentation/admin-guide/kernel-parameters.txt +4
-0@@ -3170,6 +3170,11 @@ This can be set from sysctl after boot. See Documentation/admin-guide/sysctl/vm.rst for details. + of_devlink [KNL] Make device links from common DT bindings. Useful + for optimizing probe order and making sure resources + aren't turned off before the consumer devices have + probed.
of_supplier_depend instead of of_devlink ????
of_supplier_depend
[OF, KNL] Make device links from consumer devicetree nodes to supplier devicetree nodes. The consumer / supplier relationships are inferred from scanning the devicetree. The driver for a consumer device will not be probed until the drivers for all of its supplier devices have been successfully probed.
Quoted 9 lines
Documentation/admin-guide/kernel-parameters.txt +1
-0+ ohci1394_dma=early [HW] enable debugging via the ohci1394 driver. See Documentation/debugging-via-ohci1394.txt for more info.drivers/of/platform.c +0
-0
drivers/of/platform.c int of_platform_default_populate(struct device_node *root, · line 512
Quoted 5 lines
drivers/of/platform.c +1
-0@@ -508,6 +508,170 @@ int of_platform_default_populate(struct device_node *root, }EXPORT_SYMBOL_GPL (of_platform_default_populate); +bool of_link_is_valid (struct device_node *con,struct device_node *sup)
Change to less vague:
bool of_ancestor_of(struct device_node *test_np, struct device_node *np)
Quoted 5 lines
drivers/of/platform.c +5
-0+{ +of_node_get (sup); +/* +* Don't allow linking a device node as a consumer of one of its +* descendant nodes. By definition, a child node can't be a functional
drivers/of/platform.c int of_platform_default_populate(struct device_node *root, · line 526
Quoted 10 lines
drivers/of/platform.c +10
-0+* dependency for the parent node. +*/ +while (sup) { +if (sup == con) { +of_node_put (sup); +return false; + } + sup =of_get_next_parent (sup); + } +return true;
Change to more generic:
of_node_get(test_np);
while (test_np) {
if (test_np == np) {
of_node_put(test_np);
return true;
}
test_np = of_get_next_parent(test_np);
}
return false;drivers/of/platform.c int of_platform_default_populate(struct device_node *root, · line 529
Quoted 2 lines
drivers/of/platform.c +2
-0+} +
/**
* of_link_to_phandle - Add device link to supplier
* @dev: consumer device
* @sup_np: pointer to the supplier device tree node
*
* TODO: ...
*
* Return:
* * 0 if link successfully created for supplier or of_devlink is false
* * an error if unable to create link
*/
Should have dev_debug() or pr_warn() or something on errors in this function -- the caller does not report any issue
drivers/of/platform.c int of_platform_default_populate(struct device_node *root, · line 540
Quoted 13 lines
drivers/of/platform.c +12
-0+static int of_link_to_phandle (struct device *dev,struct device_node *sup_np) +{ +struct platform_device *sup_dev; +u32 dl_flags =DL_FLAG_AUTOPROBE_CONSUMER ; +int ret =0 ; + +/* +* Since we are trying to create device links, we need to find +* the actual device node that owns this supplier phandle. +* Often times it's the same node, but sometimes it can be one +* of the parents. So walk up the parent till you find a +* device.
Change comment to:
* Find the device node that contains the supplier phandle. It may
* be @sup_np or it may be an ancestor of @sup_np.
drivers/of/platform.c int of_platform_default_populate(struct device_node *root, · line 543
Quoted 1 line
drivers/of/platform.c +1
-0+*/
See comment in caller of of_link_to_phandle() - do not hide the final of_node_put() of sup_np inside of_link_to_phandle(), so need to do an of_node_get() here.
of_node_get(sup_np);
drivers/of/platform.c int of_platform_default_populate(struct device_node *root, · line 545
Quoted 5 lines
drivers/of/platform.c +4
-0+while (sup_np && !of_find_property (sup_np,"compatible" ,NULL )) + sup_np =of_get_next_parent (sup_np); +if (!sup_np) +return 0 ;
This case should never occur(?), it is an error.
return -ENODEV;
drivers/of/platform.c int of_platform_default_populate(struct device_node *root, · line 549
Quoted 4 lines
drivers/of/platform.c +4
-0+ +if (!of_link_is_valid (dev->of_node, sup_np)) { +of_node_put (sup_np); +return 0 ;
Do not use a name that obscures what the function is doing, also return an actual issue.
if (of_ancestor_of(sup_np, dev->of_node)) {
of_node_put(sup_np);
return -EINVAL;drivers/of/platform.c int of_platform_default_populate(struct device_node *root, · line 560
Quoted 11 lines
drivers/of/platform.c +11
-0+ } + sup_dev =of_find_device_by_node (sup_np); +of_node_put (sup_np); +if (!sup_dev) +return -ENODEV ; +if (!device_link_add (dev, &sup_dev->dev, dl_flags)) + ret = -ENODEV ; +put_device (&sup_dev->dev); +return ret; +} +
/**
* parse_prop_cells - Property parsing functions for suppliers *
* @np: pointer to a device tree node containing a list
* @prop_name: Name of property holding a phandle value
* @phandle_index: For properties holding a table of phandles, this is the
* index into the table * @list_name: property name that contains a list * @cells_name: property name that specifies phandles' arguments count
*
* This function is useful to parse lists of phandles and their arguments. *
* Return:
* * Node pointer with refcount incremented, use of_node_put() on it when done.
* * NULL if not found.
*/
drivers/of/platform.c int of_platform_default_populate(struct device_node *root, · line 564
Quoted 4 lines
drivers/of/platform.c +4
-0+static struct device_node *parse_prop_cells (struct device_node *np, +const char *prop,int index, +const char *binding, +const char *cell)
Make names consistent with of_parse_phandle_with_args():
Change prop to prop_name
Change index to phandle_index
Change binding to list_name
Change cell to cells_name
drivers/of/platform.c int of_platform_default_populate(struct device_node *root, · line 570
Quoted 7 lines
drivers/of/platform.c +6
-0+{ +struct of_phandle_args sup_args; + +/* Don't need to check property name for every index. */ +if (!index &&strcmp (prop, binding)) +return NULL ;
I read the discussion on whether to check property name only once in version 6.
This check is fragile, depending upon the calling code to be properly structured. Do the check for all values of index. The reduction of overhead from not checking does not justify the fragileness and the extra complexity for the code reader to understand why the check can be bypassed when
index is not zero.
drivers/of/platform.c int of_platform_default_populate(struct device_node *root, · line 579
Quoted 9 lines
drivers/of/platform.c +9
-0+ +if (of_parse_phandle_with_args (np, binding, cell, index, &sup_args)) +return NULL ; + +return sup_args.np; +} + +static struct device_node *parse_clocks (struct device_node *np, +const char *prop,int index)
Change prop to prop_name
Change index to phandle_index
drivers/of/platform.c int of_platform_default_populate(struct device_node *root, · line 585
Quoted 6 lines
drivers/of/platform.c +6
-0+{ +return parse_prop_cells (np, prop, index,"clocks" ,"#clock-cells" ); +} + +static struct device_node *parse_interconnects (struct device_node *np, +const char *prop,int index)
Change prop to prop_name
Change index to phandle_index
Quoted 8 lines
drivers/of/platform.c +8
-0+{ +return parse_prop_cells (np, prop, index,"interconnects" , +"#interconnect-cells" ); +} + +static int strcmp_suffix (const char *str,const char *suffix) +{ +unsigned int len, suffix_len;
drivers/of/platform.c int of_platform_default_populate(struct device_node *root, · line 603
Quoted 10 lines
drivers/of/platform.c +10
-0+ + len =strlen (str); + suffix_len =strlen (suffix); +if (len <= suffix_len) +return -1 ; +return strcmp (str + len - suffix_len, suffix); +} + +static struct device_node *parse_regulators (struct device_node *np, +const char *prop,int index)
Change prop to prop_name
Change index to phandle_index
Quoted 4 lines
drivers/of/platform.c +4
-0+{ +if (index ||strcmp_suffix (prop,"-supply" )) +return NULL ; +
drivers/of/platform.c int of_platform_default_populate(struct device_node *root, · line 616
Quoted 10 lines
drivers/of/platform.c +9
-0+return of_parse_phandle (np, prop,0 ); +} + +/** +* struct supplier_bindings - Information for parsing supplier DT binding +* +* @parse_prop: If the function cannot parse the property, return NULL. +* Otherwise, return the phandle listed in the property +* that corresponds to the index.
There is no documentation of dynamic function parameters in the docbook description of a struct. Use this format for now and I will clean up when I clean up all of the devicetree docbook info.
Change above comment to:
* struct supplier_bindings - Property parsing functions for suppliers *
* @parse_prop: function name
* parse_prop() finds the node corresponding to a supplier phandle
* @parse_prop.np: Pointer to device node holding supplier phandle property
* @parse_prop.prop_name: Name of property holding a phandle value
* @parse_prop.index: For properties holding a table of phandles, this is the
* index into the table
*
* Return:
* * parse_prop() return values are
* * Node pointer with refcount incremented, use of_node_put() on it when done.
* * NULL if not found.
drivers/of/platform.c int of_platform_default_populate(struct device_node *root, · line 620
Quoted 4 lines
+ */
drivers/of/platform.c +3
-0+struct supplier_bindings { +struct device_node *(*parse_prop)(struct device_node *np, +const char *name,int index);
Change name to prop_name
Change index to phandle_index
drivers/of/platform.c int of_platform_default_populate(struct device_node *root, · line 627
Quoted 8 lines
drivers/of/platform.c +7
-0+}; + +static const struct supplier_bindings bindings[] = { + { .parse_prop = parse_clocks, }, + { .parse_prop = parse_interconnects, }, + { .parse_prop = parse_regulators, }, + { },
{},drivers/of/platform.c
Quoted 2 lines
drivers/of/platform.c +2
-0+}; +
/**
* of_link_property - TODO:
* dev:
* con_np:
* prop:
*
* TODO...
*
* Any failed attempt to create a link will NOT result in an immediate return.
* of_link_property() must create all possible links even when one of more
* attempts to create a link fail.
Why? isn't one failure enough to prevent probing this device? Continuing to scan just results in extra work... which will be repeated every time device_link_check_waiting_consumers() is called
*
* Return:
* * 0 if TODO:
* * -ENODEV on error
*/
I left some "TODO:" sections to be filled out above.
drivers/of/platform.c int of_platform_default_populate(struct device_node *root, · line 631
Quoted 2 lines
drivers/of/platform.c +2
-0+static bool of_link_property (struct device *dev,struct device_node *con_np, +const char *prop)
Returns 0 or -ENODEV, so bool is incorrect
(Also fixed on 8/8 in patch: "[PATCH 1/2] of/platform: Fix fn definitons for of_link_is_valid() and of_link_property()")
drivers/of/platform.c int of_platform_default_populate(struct device_node *root, · line 636
Quoted 6 lines
drivers/of/platform.c +5
-0+{ +struct device_node *phandle; +struct supplier_bindings *s = bindings; +unsigned int i =0 ; +bool done = true, matched = false;
Change to:
bool matched = false; int ret = 0; /* do not stop at first failed link, link all available suppliers */
drivers/of/platform.c int of_platform_default_populate(struct device_node *root, · line 642
Quoted 6 lines
drivers/of/platform.c +6
-0+ +while (!matched && s->parse_prop) { +while ((phandle = s->parse_prop (con_np, prop, i))) { + matched = true; + i++; +if (of_link_to_phandle (dev, phandle))
Remove comment:
drivers/of/platform.c int of_platform_default_populate(struct device_node *root, · line 648
Quoted 7 lines
drivers/of/platform.c +6
-0+/* +* Don't stop at the first failure. See +* Documentation for bus_type.add_links for +* more details. +*/ + done = false;
ret = -ENODEV;
Do not hide of_node_put() inside of_link_to_phandle(), do it here:
of_node_put(phandle);
drivers/of/platform.c int of_platform_default_populate(struct device_node *root, · line 652
Quoted 5 lines
drivers/of/platform.c +4
-0+ } + s++; + } +return done ?0 : -ENODEV ;
return ret;
drivers/of/platform.c int of_platform_default_populate(struct device_node *root, · line 657
Quoted 5 lines
drivers/of/platform.c +5
-0+} + +static bool of_devlink; +core_param (of_devlink, of_devlink, bool,0 ); +
/**
* of_link_to_suppliers - Add device links to suppliers
* @dev: consumer device
*
* Create device links to all available suppliers of @dev.
* Must NOT stop at the first failed link.
* If some suppliers are not yet available, this function will be
* called again when additional suppliers become available. *
* Return:
* * 0 if links successfully created for all suppliers
* * an error if one or more suppliers not yet available */
drivers/of/platform.c int of_platform_default_populate(struct device_node *root, · line 661
Quoted 5 lines
drivers/of/platform.c +4
-0+static int of_link_to_suppliers (struct device *dev) +{ +struct property *p; +bool done = true;
remove done
int ret = 0;
drivers/of/platform.c int of_platform_default_populate(struct device_node *root, · line 666
Quoted 6 lines
drivers/of/platform.c +5
-0+ +if (!of_devlink) +return 0 ; +if (unlikely (!dev->of_node)) +return 0 ;
Check not needed, for_each_property_of_node() will detect !dev->of_node.
drivers/of/platform.c int of_platform_default_populate(struct device_node *root, · line 670
Quoted 5 lines
drivers/of/platform.c +4
-0+ +for_each_property_of_node (dev->of_node, p) +if (of_link_property (dev, dev->of_node, p->name)) + done = false;
ret = -EAGAIN;
drivers/of/platform.c
Quoted 2 lines
drivers/of/platform.c +2
-0+ +return done ?0 : -ENODEV ;
return ret;
Quoted 17 lines
drivers/of/platform.c +3
-0+} +#ifndef CONFIG_PPC static const struct of_device_id reserved_mem_matches[] = { { .compatible = "qcom,rmtfs-mem" },@@ -523,6 +687,7 @@ static int __init of_platform_default_populate_init(void)if (!of_have_populated_dt ())return -ENODEV ; + platform_bus_type.add_links = of_link_to_suppliers;/* * Handle certain compatibles explicitly, since we don't want to create * platform_devices for every node in /reserved-memory with a Signature · 1 line
2.22.0.709.g102302147b-goog
Re: [PATCH v7 3/7] of/platform: Add functional dependency link from DT bindings
Quoted 54 lines · Frank Rowand
Quoted 45 lines · Saravana Kannan
Add device-links after the devices are created (but before they are probed) by looking at common DT bindings like clocks and interconnects.
Automatically adding device-links for functional dependencies at the framework level provides the following benefits:
- Optimizes device probe order and avoids the useless work of attempting probes of devices that will not probe successfully (because their suppliers aren't present or haven't probed yet).
For example, in a commonly available mobile SoC, registering just one consumer device's driver at an initcall level earlier than the supplier device's driver causes 11 failed probe attempts before the consumer device probes successfully. This was with a kernel with all the drivers statically compiled in. This problem gets a lot worse if all the drivers are loaded as modules without direct symbol dependencies.
- Supplier devices like clock providers, interconnect providers, etc need to keep the resources they provide active and at a particular state(s) during boot up even if their current set of consumers don't request the resource to be active. This is because the rest of the consumers might not have probed yet and turning off the resource before all the consumers have probed could lead to a hang or undesired user experience.
Some frameworks (Eg: regulator) handle this today by turning off "unused" resources at
late_initcall_syncand hoping all the devices have probed by then. This is not a valid assumption for systems with loadable modules. Other frameworks (Eg: clock) just don't handle this due to the lack of a clear signal for when they can turn off resources. This leads to downstream hacks to handle cases like this that can easily be solved in the upstream kernel.By linking devices before they are probed, we give suppliers a clear count of the number of dependent consumers. Once all of the consumers are active, the suppliers can turn off the unused resources without making assumptions about the number of consumers.
By default we just add device-links to track "driver presence" (probe succeeded) of the supplier device. If any other functionality provided by device-links are needed, it is left to the consumer/supplier devices to change the link when they probe.
Quoted 6 lines
Signed-off-by Saravana Kannan <[email protected]>--- .../admin-guide/kernel-parameters.txt | 5 + drivers/of/platform.c | 165 ++++++++++++++++++ 2 files changed, 170 insertions(+)
Quoted 5 lines
Documentation/admin-guide/kernel-paramers.rst:After line 129, add:
OF Devicetree is enabled
Will do.
Quoted 8 lines
Quoted 8 lines
Documentation/admin-guide/kernel-parameters.txt +0
-0@@ -3170,6 +3170,11 @@ This can be set from sysctl after boot. See Documentation/admin-guide/sysctl/vm.rst for details.
Quoted 6 lines
Documentation/admin-guide/kernel-parameters.txt line 3176
Quoted 4 lines
Documentation/admin-guide/kernel-parameters.txt +4
-0+ of_devlink [KNL] Make device links from common DT bindings. Useful + for optimizing probe order and making sure resources + aren't turned off before the consumer devices have + probed.
of_supplier_dependinstead ofof_devlink????
I'm open to other names, but of_supplier_depend is just odd. of_devlink stands for of_device_links. If someone wants to know what device links do, they can read up on the device links documentation?
Quoted 4 lines
of_supplier_depend
[OF, KNL] Make device links from consumer devicetree nodes to supplier devicetree nodes.
We are creating device links between devices. Not device tree nodes. So this would be wrong. I'll replace it with "devicetree nodes" with "devices"?
Quoted 3 lines
The
consumer / supplier relationships are inferred from scanning the devicetree.
I like this part. How about clarifying it with "from scanning the common bindings in the devicetree"? Because we aren't trying to scan the device specific properties.
Quoted 3 lines
The driver for a consumer
device will not be probed until the drivers for all of its supplier devices have been successfully probed.
A driver is never probed. It's a device that's probed. The dependency is between devices and not drivers. So, how about I replace this with:
"The consumer device will not be probed until all of its supplier devices are successfully probed"?
Also, any reason you removed the "resources aren't turned off ..." part? That's one of the biggest improvements of this patch series. Do you want to rewrite that part too? Or I'll leave my current wording of that part as is?
Quoted 15 lines
Quoted 14 lines
Documentation/admin-guide/kernel-parameters.txt +1
-0+ ohci1394_dma=early [HW] enable debugging via the ohci1394 driver. See Documentation/debugging-via-ohci1394.txt for more info.drivers/of/platform.c +0
-0@@ -508,6 +508,170 @@ int of_platform_default_populate(struct device_node *root, }EXPORT_SYMBOL_GPL (of_platform_default_populate);
Quoted 5 lines
drivers/of/platform.c int of_platform_default_populate(struct device_node *root, · line 511
Quoted 1 line
drivers/of/platform.c +1
-0+bool of_link_is_valid (struct device_node *con,struct device_node *sup)Change to less vague:
bool
of_ancestor_of(struct device_node *test_np, struct device_node *np)
I thought about that when I wrote the code. But the consumer being the ancestor of the supplier is just one of the things that makes a link invalid. There can be other tests we might add as we go. That's why I kept the function name as of_link_is_valid(). Even if I add an "ancestor_of" helper function, I'll still have of_link_is_valid() as a wrapper around it.
Quoted 18 lines
Quoted 5 lines
drivers/of/platform.c +5
-0+{ +of_node_get (sup); +/* +* Don't allow linking a device node as a consumer of one of its +* descendant nodes. By definition, a child node can't be a functional drivers/of/platform.c int of_platform_default_populate(struct device_node *root, · line 526
Quoted 10 lines
drivers/of/platform.c +10
-0+* dependency for the parent node. +*/ +while (sup) { +if (sup == con) { +of_node_put (sup); +return false; + } + sup =of_get_next_parent (sup); + } +return true;Change to more generic:
Quoted 9 lines
of_node_get(test_np);
while (test_np) {
if (test_np == np) {
of_node_put(test_np);
return true;
}
test_np = of_get_next_parent(test_np);
}
return false;If you do insist on this change, I think we need better names than test_np and np. It's not clear which node needs to be the ancestor for this function to return true. With consumer/supplier, it's kinda obvious that a consumer can't be an ancestor of a supplier.
Quoted 8 lines
drivers/of/platform.c int of_platform_default_populate(struct device_node *root, · line 528
Quoted 2 lines
drivers/of/platform.c +2
-0+} +/**
*of_link_to_phandle- Add device link to supplier
* @dev: consumer device
* @sup_np: pointer to the supplier device tree node
Could you suggest something that makes it a bit more clear that this phandle/node doesn't need to be an actual device (as in, one with compatible property)? phandle kinda make it clear at least to me. I'd prefer just saying "phandle to supplier". Thoughts?
Also, what's the guideline on which functions needs doc headers? I always leaned towards adding them only for non-static functions. That's why I skipped this one. What's the reasoning for needing one for this? I'm happy to do this, just want to see that there's some consistent guideline that's being followed and something I can use for future patches.
Quoted 5 lines
*
* TODO: ...
*
* Return:
* * 0 if link successfully created for supplier orof_devlinkis false
The "or of_devlink is false" isn't true for this function?
Quoted 5 lines
* * an error if unable to create link
*/Should have
dev_debug()orpr_warn()or something on errors in this function -- the caller does not report any issue
I think that'll be too spammy during bootup. This function is expected to fail often and it's not necessary a catastrophic failure. The caller can print something if they care to. The current set of callers don't.
Quoted 14 lines
drivers/of/platform.c int of_platform_default_populate(struct device_node *root, · line 540
Quoted 13 lines
drivers/of/platform.c +12
-0+static int of_link_to_phandle (struct device *dev,struct device_node *sup_np) +{ +struct platform_device *sup_dev; +u32 dl_flags =DL_FLAG_AUTOPROBE_CONSUMER ; +int ret =0 ; + +/* +* Since we are trying to create device links, we need to find +* the actual device node that owns this supplier phandle. +* Often times it's the same node, but sometimes it can be one +* of the parents. So walk up the parent till you find a +* device.
Quoted 4 lines
Change comment to:
* Find the device node that contains the supplier phandle. It may
* be @sup_np or it may be an ancestor of @sup_np.
Aren't the existing comments giving a better explanation with more context? But I'll do this.
Quoted 8 lines
drivers/of/platform.c int of_platform_default_populate(struct device_node *root, · line 542
Quoted 2 lines
drivers/of/platform.c +1
-0+*/ See comment in caller of
of_link_to_phandle()- do not hide the finalof_node_put()ofsup_npinsideof_link_to_phandle(), so need to do anof_node_get()here.of_node_get(sup_np);
Will do. Good point.
Quoted 10 lines
drivers/of/platform.c int of_platform_default_populate(struct device_node *root, · line 545
Quoted 6 lines
drivers/of/platform.c +4
-0+while (sup_np && !of_find_property (sup_np,"compatible" ,NULL )) + sup_np =of_get_next_parent (sup_np); +if (!sup_np) +return 0 ;This case should never occur(?), it is an error.
return -ENODEV;
I'm not too sure about all the possible DT combinations to say this? In that case, this isn't an error. As in, the consumer doesn't need to wait for this non-existent device to get populated/added. I'd lean towards leaving it as is and address it later if this is actually a problem. I want of_link_to_phandle to fail only when a link can be created, but isn't due to current system state (device isn't added, creating a cyclic link, etc).
Quoted 6 lines
drivers/of/platform.c int of_platform_default_populate(struct device_node *root, · line 549
Quoted 5 lines
drivers/of/platform.c +4
-0+ +if (!of_link_is_valid (dev->of_node, sup_np)) { +of_node_put (sup_np); +return 0 ;
Quoted 6 lines
Do not use a name that obscures what the function is doing, also return an actual issue.
if (of_ancestor_of(sup_np, dev->of_node)) { of_node_put(sup_np); return -EINVAL;
See my comment about not erroring on cases where a link can't ever be created? So in your case, you'd want the caller to check the error value to device which ones to ignore and which ones not to? That seems a bit more fragile when this function is potentially changed in the furture.
Quoted 8 lines
drivers/of/platform.c int of_platform_default_populate(struct device_node *root, · line 556
Quoted 8 lines
drivers/of/platform.c +7
-0+ } + sup_dev =of_find_device_by_node (sup_np); +of_node_put (sup_np); +if (!sup_dev) +return -ENODEV ; +if (!device_link_add (dev, &sup_dev->dev, dl_flags)) + ret = -ENODEV ;
For example, in the earlier comment you suggested -ENODEV if there's no device with "compatible" property that encapsulates the supplier phandle. But -ENODEV makes more sense for this case where there's actually no device because it hasn't been added yet. And the caller needs to be able to distinguish between these two. Are we just going to arbitrarily pick error values just to make sure they don't overlap?
I don't have a strong opinion one way or another, but I'm trying to understand what's better in the long run where this function can evolve to add more checks or handle more cases.
Quoted 22 lines
drivers/of/platform.c int of_platform_default_populate(struct device_node *root, · line 560
Quoted 4 lines
drivers/of/platform.c +4
-0+put_device (&sup_dev->dev); +return ret; +} +/**
*parse_prop_cells- Property parsing functions for suppliers *
* @np: pointer to a device tree node containing a list
* @prop_name: Name of property holding a phandle value
* @phandle_index: For properties holding a table of phandles, this is the* index into the table * @list_name: property name that contains a list * @cells_name: property name that specifies phandles' arguments count*
* This function is useful to parse lists of phandles and their arguments. *
* Return:
* * Node pointer with refcount incremented, useof_node_put()on it when done.
* * NULL if not found.
*/
Quoted 8 lines
drivers/of/platform.c int of_platform_default_populate(struct device_node *root, · line 564
Quoted 4 lines
drivers/of/platform.c +4
-0+static struct device_node *parse_prop_cells (struct device_node *np, +const char *prop,int index, +const char *binding, +const char *cell)Make names consistent with
of_parse_phandle_with_args():
Change prop toprop_name
Change index tophandle_index
You call this index even in of_parse_phandle_with_args()
Quoted 2 lines
Change binding to
list_name
Change cell tocells_name
This going to cause a lot more line wraps for barely better names. But I'll reluctantly do this.
Quoted 12 lines
drivers/of/platform.c int of_platform_default_populate(struct device_node *root, · line 570
Quoted 8 lines
drivers/of/platform.c +6
-0+{ +struct of_phandle_args sup_args; + +/* Don't need to check property name for every index. */ +if (!index &&strcmp (prop, binding)) +return NULL ;I read the discussion on whether to check property name only once in version 6.
Quoted 6 lines
This check is fragile, depending upon the calling code to be properly structured. Do the check for all values of index. The reduction of overhead from not checking does not justify the fragileness and the extra complexity for the code reader to understand why the check can be bypassed when
index is not zero.
This is used only in this file. I understand needing the balance between code complexity/fragility and efficiency. But I think you push the line too far away from efficiency. This code is literally never going to fail because it's a static function called only inside this file. And the check isn't that hard to understand with that tiny comment.
Quoted 36 lines
drivers/of/platform.c int of_platform_default_populate(struct device_node *root, · line 579
Quoted 9 lines
drivers/of/platform.c +9
-0+ +if (of_parse_phandle_with_args (np, binding, cell, index, &sup_args)) +return NULL ; + +return sup_args.np; +} + +static struct device_node *parse_clocks (struct device_node *np, +const char *prop,int index)Change prop to
prop_name
Change index tophandle_indexdrivers/of/platform.c int of_platform_default_populate(struct device_node *root, · line 585
Quoted 6 lines
drivers/of/platform.c +6
-0+{ +return parse_prop_cells (np, prop, index,"clocks" ,"#clock-cells" ); +} + +static struct device_node *parse_interconnects (struct device_node *np, +const char *prop,int index)Change prop to
prop_name
Change index tophandle_indexQuoted 13 lines
drivers/of/platform.c +13
-0+{ +return parse_prop_cells (np, prop, index,"interconnects" , +"#interconnect-cells" ); +} + +static int strcmp_suffix (const char *str,const char *suffix) +{ +unsigned int len, suffix_len; + + len =strlen (str); + suffix_len =strlen (suffix); +if (len <= suffix_len) +return -1 ;
Quoted 9 lines
drivers/of/platform.c int of_platform_default_populate(struct device_node *root, · line 603
Quoted 5 lines
drivers/of/platform.c +5
-0+return strcmp (str + len - suffix_len, suffix); +} + +static struct device_node *parse_regulators (struct device_node *np, +const char *prop,int index)Change prop to
prop_name
Change index tophandle_index
Will do all the renames you list above.
Quoted 21 lines
Quoted 4 lines
drivers/of/platform.c +4
-0+{ +if (index ||strcmp_suffix (prop,"-supply" )) +return NULL ; +drivers/of/platform.c int of_platform_default_populate(struct device_node *root, · line 616
Quoted 10 lines
drivers/of/platform.c +9
-0+return of_parse_phandle (np, prop,0 ); +} + +/** +* struct supplier_bindings - Information for parsing supplier DT binding +* +* @parse_prop: If the function cannot parse the property, return NULL. +* Otherwise, return the phandle listed in the property +* that corresponds to the index. There is no documentation of dynamic function parameters in the docbook description of a struct. Use this format for now and I will clean up when I clean up all of the devicetree docbook info.
Change above comment to:
Quoted 13 lines
* struct
supplier_bindings- Property parsing functions for suppliers *
* @parse_prop: function name
*parse_prop()finds the node corresponding to a supplier phandle
* @parse_prop.np: Pointer to device node holding supplier phandle property
* @parse_prop.prop_name: Name of property holding a phandle value
* @parse_prop.index: For properties holding a table of phandles, this is the
* index into the table
*
* Return:
* *parse_prop()return values are
* * Node pointer with refcount incremented, useof_node_put()on it when done.
* * NULL if not found.
Will do. Thanks for writing it.
Quoted 34 lines
drivers/of/platform.c int of_platform_default_populate(struct device_node *root, · line 620
Quoted 4 lines
+ */
drivers/of/platform.c +3
-0+struct supplier_bindings { +struct device_node *(*parse_prop)(struct device_node *np, +const char *name,int index);Change name to
prop_name
Change index tophandle_indexdrivers/of/platform.c int of_platform_default_populate(struct device_node *root, · line 627
Quoted 8 lines
drivers/of/platform.c +7
-0+}; + +static const struct supplier_bindings bindings[] = { + { .parse_prop = parse_clocks, }, + { .parse_prop = parse_interconnects, }, + { .parse_prop = parse_regulators, }, + { },{},drivers/of/platform.c
Quoted 2 lines
drivers/of/platform.c +2
-0+}; +/**
*of_link_property- TODO:
* dev:
*con_np:
* prop:
*
* TODO...
*
* Any failed attempt to create a link will NOT result in an immediate return.
*of_link_property()must create all possible links even when one of more
* attempts to create a link fail.
Quoted 3 lines
Why? isn't one failure enough to prevent probing this device? Continuing to scan just results in extra work... which will be repeated every time
device_link_check_waiting_consumers()is called
Context:
As I said in the cover letter, avoiding unnecessary probes is just one of the reasons for this patch. The other (arguably more important) reason for this patch is to make sure suppliers know that they have consumers that are yet to be probed. That way, suppliers can leave their resource on AND in the right state if they were left on by the bootloader. For example, if a clock was left on and at 200 MHz, the clock provider needs to keep that clock ON and at 200 MHz till all the consumers are probed.
Answer: Let's say a consumer device Z has suppliers A, B and C. If the linking fails at A and you return immediately, then B and C could probe and then figure that they have no more consumers (they don't see a link to Z) and turn off their resources. And Z could fail catastrophically.
Quoted 8 lines
*
* Return:
* * 0 if TODO:
* *-ENODEVon error
*/I left some "TODO:" sections to be filled out above.
Will do.
Quoted 9 lines
drivers/of/platform.c int of_platform_default_populate(struct device_node *root, · line 631
Quoted 4 lines
drivers/of/platform.c +2
-0+static bool of_link_property (struct device *dev,struct device_node *con_np, +const char *prop)Returns 0 or
-ENODEV, so bool is incorrect(Also fixed on 8/8 in patch: "[PATCH 1/2] of/platform: Fix fn definitons for
of_link_is_valid()andof_link_property()")
Right.
Quoted 23 lines
drivers/of/platform.c int of_platform_default_populate(struct device_node *root, · line 636
Quoted 7 lines
drivers/of/platform.c +5
-0+{ +struct device_node *phandle; +struct supplier_bindings *s = bindings; +unsigned int i =0 ; +bool done = true, matched = false;Change to:
bool matched = false; int ret = 0; /* do not stop at first failed link, link all available suppliers */drivers/of/platform.c int of_platform_default_populate(struct device_node *root, · line 642
Quoted 6 lines
drivers/of/platform.c +6
-0+ +while (!matched && s->parse_prop) { +while ((phandle = s->parse_prop (con_np, prop, i))) { + matched = true; + i++; +if (of_link_to_phandle (dev, phandle))
Quoted 7 lines
Remove comment:
Quoted 5 lines
drivers/of/platform.c +5
-0+/* +* Don't stop at the first failure. See +* Documentation for bus_type.add_links for +* more details. +*/
Ok
Quoted 4 lines
Quoted 2 lines
+ done = false;
ret = -ENODEV;
This is nicer. Thanks.
Quoted 4 lines
Do not hide
of_node_put()insideof_link_to_phandle(), do it here:of_node_put(phandle);
Ok
Quoted 20 lines
drivers/of/platform.c int of_platform_default_populate(struct device_node *root, · line 652
Quoted 6 lines
drivers/of/platform.c +4
-0+ } + s++; + } +return done ?0 : -ENODEV ;return ret;drivers/of/platform.c int of_platform_default_populate(struct device_node *root, · line 657
Quoted 5 lines
drivers/of/platform.c +5
-0+} + +static bool of_devlink; +core_param (of_devlink, of_devlink, bool,0 ); +/**
*of_link_to_suppliers- Add device links to suppliers
* @dev: consumer device
*
* Create device links to all available suppliers of @dev.
Quoted 8 lines
* Must NOT stop at the first failed link.
* If some suppliers are not yet available, this function will be
* called again when additional suppliers become available. *
* Return:
* * 0 if links successfully created for all suppliers
* * an error if one or more suppliers not yet available */
Ok
Quoted 23 lines
drivers/of/platform.c int of_platform_default_populate(struct device_node *root, · line 661
Quoted 5 lines
drivers/of/platform.c +4
-0+static int of_link_to_suppliers (struct device *dev) +{ +struct property *p; +bool done = true;remove done
int ret = 0;drivers/of/platform.c int of_platform_default_populate(struct device_node *root, · line 666
Quoted 6 lines
drivers/of/platform.c +5
-0+ +if (!of_devlink) +return 0 ; +if (unlikely (!dev->of_node)) +return 0 ;Check not needed,
for_each_property_of_node()will detect !dev->of_node.Quoted 3 lines
drivers/of/platform.c +3
-0+ +for_each_property_of_node (dev->of_node, p) +if (of_link_property (dev, dev->of_node, p->name))
Quoted 8 lines
Quoted 1 line
+ done = false;
ret = -EAGAIN;drivers/of/platform.c
Quoted 2 lines
drivers/of/platform.c +2
-0+ +return done ?0 : -ENODEV ;return ret;
Thanks. I think I was too caught up on the rest of the logic complexity that I missed this obviously ugly code. Will fix.
Thanks,
Saravana
Re: [PATCH v7 3/7] of/platform: Add functional dependency link from DT bindings
Quoted 6 lines · Saravana Kannan
Quoted 5 lines · Frank Rowand
Quoted 3 lines · Saravana Kannan
Add device-links after the devices are created (but before they are probed) by looking at common DT bindings like clocks and interconnects.
< very big snip (lots of comments that deserve answers) >
Quoted 13 lines
Quoted 13 lines
/**
*of_link_property- TODO:
* dev:
*con_np:
* prop:
*
* TODO...
*
* Any failed attempt to create a link will NOT result in an immediate return.
*of_link_property()must create all possible links even when one of more
* attempts to create a link fail.
Quoted 7 lines
Quoted 3 lines
Why? isn't one failure enough to prevent probing this device? Continuing to scan just results in extra work... which will be repeated every time
device_link_check_waiting_consumers()is calledContext:
As I said in the cover letter, avoiding unnecessary probes is just one of the reasons for this patch. The other (arguably more important)
Agree that it is more important.
Quoted 7 lines
reason for this patch is to make sure suppliers know that they have consumers that are yet to be probed. That way, suppliers can leave their resource on AND in the right state if they were left on by the bootloader. For example, if a clock was left on and at 200 MHz, the clock provider needs to keep that clock ON and at 200 MHz till all the consumers are probed.
Quoted 5 lines
Answer: Let's say a consumer device Z has suppliers A, B and C. If the linking fails at A and you return immediately, then B and C could probe and then figure that they have no more consumers (they don't see a link to Z) and turn off their resources. And Z could fail catastrophically.
Then I think that this approach is fatally flawed in the current implementation.
A device can be added by a module that is loaded. In that case the device was not present at late boot when the suppliers may turn off their resources. (I am assuming the details since I have not reviewed the patches later in the series that implement this part.)
Am I missing something?
If I am wrong, then I'll have more comments for your review replies for patches 2 and 3.
Quoted 1 line
< another snip >
Quoted 3 lines
Thanks,
Saravana
-Frank
Re: [PATCH v7 3/7] of/platform: Add functional dependency link from DT bindings
Quoted 44 lines · Frank Rowand
Quoted 6 lines · Saravana Kannan
Quoted 5 lines · Frank Rowand
Quoted 3 lines · Saravana Kannan
Add device-links after the devices are created (but before they are probed) by looking at common DT bindings like clocks and interconnects.
< very big snip (lots of comments that deserve answers) >
Quoted 13 lines
Quoted 13 lines
/**
*of_link_property- TODO:
* dev:
*con_np:
* prop:
*
* TODO...
*
* Any failed attempt to create a link will NOT result in an immediate return.
*of_link_property()must create all possible links even when one of more
* attempts to create a link fail.Quoted 7 lines
Quoted 3 lines
Why? isn't one failure enough to prevent probing this device? Continuing to scan just results in extra work... which will be repeated every time
device_link_check_waiting_consumers()is calledContext:
As I said in the cover letter, avoiding unnecessary probes is just one of the reasons for this patch. The other (arguably more important)Agree that it is more important.
Quoted 7 lines
reason for this patch is to make sure suppliers know that they have consumers that are yet to be probed. That way, suppliers can leave their resource on AND in the right state if they were left on by the bootloader. For example, if a clock was left on and at 200 MHz, the clock provider needs to keep that clock ON and at 200 MHz till all the consumers are probed.
Quoted 7 lines
Quoted 5 lines
Answer: Let's say a consumer device Z has suppliers A, B and C. If the linking fails at A and you return immediately, then B and C could probe and then figure that they have no more consumers (they don't see a link to Z) and turn off their resources. And Z could fail catastrophically.
Then I think that this approach is fatally flawed in the current implementation.
I'm waiting to hear how it is fatally flawed. But maybe this is just a misunderstanding of the problem?
In the text below, I'm not sure if you mixing up two different things or just that your wording it a bit ambiguous. So pardon my nitpick to err on the side of clarity.
Quoted 1 line
A device can be added by a module that is loaded.
No, in the example I gave, of_platform_default_populate_init() would add all 3 of those devices during arch_initcall_sync().
Quoted 2 lines
In that case the device
was not present at late boot when the suppliers may turn off their resources.
In that case, the _drivers_ for those devices aren't present at late boot. So that they can't request to keep the resources on for their consumer devices. Since there are no consumer requests on resources, the suppliers turn off their resources at late boot (since there isn't a better location as of today). The sync_state() call back added in a subsequent patche in this series will provide the better location.
Quoted 4 lines
(I am assuming the details since I have not reviewed the patches later in the series that implement this part.)
Am I missing something?
I think you are mixing up devices getting added/populated with drivers getting loaded as modules?
Quoted 2 lines
If I am wrong, then I'll have more comments for your review replies for patches 2 and 3.
I'll wait for more review replies?
Thanks,
Saravana
Re: [PATCH v7 3/7] of/platform: Add functional dependency link from DT bindings
Quoted 51 lines · Saravana Kannan
Quoted 50 lines · Frank Rowand
Quoted 6 lines · Saravana Kannan
Quoted 5 lines · Frank Rowand
Quoted 3 lines · Saravana Kannan
Add device-links after the devices are created (but before they are probed) by looking at common DT bindings like clocks and interconnects.
< very big snip (lots of comments that deserve answers) >
Quoted 13 lines
Quoted 13 lines
/**
*of_link_property- TODO:
* dev:
*con_np:
* prop:
*
* TODO...
*
* Any failed attempt to create a link will NOT result in an immediate return.
*of_link_property()must create all possible links even when one of more
* attempts to create a link fail.Quoted 7 lines
Quoted 3 lines
Why? isn't one failure enough to prevent probing this device? Continuing to scan just results in extra work... which will be repeated every time
device_link_check_waiting_consumers()is calledContext:
As I said in the cover letter, avoiding unnecessary probes is just one of the reasons for this patch. The other (arguably more important)Agree that it is more important.
Quoted 12 lines
reason for this patch is to make sure suppliers know that they have consumers that are yet to be probed. That way, suppliers can leave their resource on AND in the right state if they were left on by the bootloader. For example, if a clock was left on and at 200 MHz, the clock provider needs to keep that clock ON and at 200 MHz till all the consumers are probed.
Answer: Let's say a consumer device Z has suppliers A, B and C. If the linking fails at A and you return immediately, then B and C could probe and then figure that they have no more consumers (they don't see a link to Z) and turn off their resources. And Z could fail catastrophically.
Quoted 4 lines
Quoted 1 line
Then I think that this approach is fatally flawed in the current implementation.
I'm waiting to hear how it is fatally flawed. But maybe this is just a misunderstanding of the problem?
Fatally flawed because it does not handle modules that add a consumer device when the module is loaded.
Quoted 4 lines
In the text below, I'm not sure if you mixing up two different things or just that your wording it a bit ambiguous. So pardon my nitpick to err on the side of clarity.
Please do nitpick. Clarity is good.
Quoted 5 lines
Quoted 2 lines
A device can be added by a module that is loaded.
No, in the example I gave,
of_platform_default_populate_init()would add all 3 of those devices duringarch_initcall_sync().
The example you gave does not cover all use cases.
There are modules that add devices when the module is loaded. You can not ignore systems using such modules.
Quoted 10 lines
Quoted 3 lines
In that case the device
was not present at late boot when the suppliers may turn off their resources.In that case, the _drivers_ for those devices aren't present at late boot. So that they can't request to keep the resources on for their consumer devices. Since there are no consumer requests on resources, the suppliers turn off their resources at late boot (since there isn't a better location as of today). The
sync_state()call back added in a subsequent patche in this series will provide the better location.
And the sync_state() call back will not deal with modules that add consumer devices when the module is loaded, correct?
Quoted 8 lines
Quoted 5 lines
(I am assuming the details since I have not reviewed the patches later in the series that implement this part.)
Am I missing something?
I think you are mixing up devices getting added/populated with drivers getting loaded as modules?
Only some modules add devices when they are loaded. But these modules do exist.
-Frank
Quoted 9 lines
Quoted 3 lines
If I am wrong, then I'll have more comments for your review replies for patches 2 and 3.
I'll wait for more review replies?
Thanks,
Saravana
Re: [PATCH v7 3/7] of/platform: Add functional dependency link from DT bindings
Quoted 53 lines · Frank Rowand
Quoted 51 lines · Saravana Kannan
Quoted 50 lines · Frank Rowand
Quoted 6 lines · Saravana Kannan
Quoted 5 lines · Frank Rowand
Quoted 3 lines · Saravana Kannan
Add device-links after the devices are created (but before they are probed) by looking at common DT bindings like clocks and interconnects.
< very big snip (lots of comments that deserve answers) >
Quoted 13 lines
Quoted 13 lines
/**
*of_link_property- TODO:
* dev:
*con_np:
* prop:
*
* TODO...
*
* Any failed attempt to create a link will NOT result in an immediate return.
*of_link_property()must create all possible links even when one of more
* attempts to create a link fail.Quoted 7 lines
Quoted 3 lines
Why? isn't one failure enough to prevent probing this device? Continuing to scan just results in extra work... which will be repeated every time
device_link_check_waiting_consumers()is calledContext:
As I said in the cover letter, avoiding unnecessary probes is just one of the reasons for this patch. The other (arguably more important)Agree that it is more important.
Quoted 12 lines
reason for this patch is to make sure suppliers know that they have consumers that are yet to be probed. That way, suppliers can leave their resource on AND in the right state if they were left on by the bootloader. For example, if a clock was left on and at 200 MHz, the clock provider needs to keep that clock ON and at 200 MHz till all the consumers are probed.
Answer: Let's say a consumer device Z has suppliers A, B and C. If the linking fails at A and you return immediately, then B and C could probe and then figure that they have no more consumers (they don't see a link to Z) and turn off their resources. And Z could fail catastrophically.
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
Then I think that this approach is fatally flawed in the current implementation.
I'm waiting to hear how it is fatally flawed. But maybe this is just a misunderstanding of the problem?
Fatally flawed because it does not handle modules that add a consumer device when the module is loaded.
If you are talking about modules adding child devices of the device they are managing, then that's handled correctly later in the series.
If you are talking about modules adding devices that aren't defined in DT, then right, I'm not trying to handle that. The module needs to make sure it keeps the resources needed for new devices it's adding are in the right state or need to add the right device links.
Quoted 10 lines
Quoted 3 lines
In the text below, I'm not sure if you mixing up two different things or just that your wording it a bit ambiguous. So pardon my nitpick to err on the side of clarity.
Please do nitpick. Clarity is good.
Quoted 3 lines
Quoted 2 lines
A device can be added by a module that is loaded.
Quoted 7 lines
Quoted 2 lines
No, in the example I gave,
of_platform_default_populate_init()would add all 3 of those devices duringarch_initcall_sync().The example you gave does not cover all use cases.
There are modules that add devices when the module is loaded. You can not ignore systems using such modules.
I'll have to agree to disagree on that. While I understand that the design should be good and I'm happy to work on that, you can't insist that a patch series shouldn't be allowed because it's only improving 99% of the cases and leaves the other 1% in the status quo. You are just going to bring the kernel development to a grinding halt.
Quoted 11 lines
Quoted 10 lines
Quoted 3 lines
In that case the device
was not present at late boot when the suppliers may turn off their resources.In that case, the _drivers_ for those devices aren't present at late boot. So that they can't request to keep the resources on for their consumer devices. Since there are no consumer requests on resources, the suppliers turn off their resources at late boot (since there isn't a better location as of today). The
sync_state()call back added in a subsequent patche in this series will provide the better location.
Quoted 2 lines
And the
sync_state()call back will not deal with modules that add consumer devices when the module is loaded, correct?
Depends. If it's just more devices from DT, then it'll be fine. If it's not, then the module needs to take care of the needs of devices it's adding.
Quoted 11 lines
Quoted 8 lines
Quoted 5 lines
(I am assuming the details since I have not reviewed the patches later in the series that implement this part.)
Am I missing something?
I think you are mixing up devices getting added/populated with drivers getting loaded as modules?
Only some modules add devices when they are loaded. But these modules do exist.
Out of the billions of Android devices, how many do you see this happening in?
Thanks,
Saravana
Re: [PATCH v7 3/7] of/platform: Add functional dependency link from DT bindings
Quoted 56 lines · Saravana Kannan
Quoted 55 lines · Frank Rowand
Quoted 53 lines · Saravana Kannan
Quoted 51 lines · Frank Rowand
Quoted 6 lines · Saravana Kannan
Quoted 5 lines · Frank Rowand
Quoted 3 lines · Saravana Kannan
Add device-links after the devices are created (but before they are probed) by looking at common DT bindings like clocks and interconnects.
< very big snip (lots of comments that deserve answers) >
Quoted 13 lines
Quoted 13 lines
/**
*of_link_property- TODO:
* dev:
*con_np:
* prop:
*
* TODO...
*
* Any failed attempt to create a link will NOT result in an immediate return.
*of_link_property()must create all possible links even when one of more
* attempts to create a link fail.Quoted 7 lines
Quoted 3 lines
Why? isn't one failure enough to prevent probing this device? Continuing to scan just results in extra work... which will be repeated every time
device_link_check_waiting_consumers()is calledContext:
As I said in the cover letter, avoiding unnecessary probes is just one of the reasons for this patch. The other (arguably more important)Agree that it is more important.
Quoted 7 lines
reason for this patch is to make sure suppliers know that they have consumers that are yet to be probed. That way, suppliers can leave their resource on AND in the right state if they were left on by the bootloader. For example, if a clock was left on and at 200 MHz, the clock provider needs to keep that clock ON and at 200 MHz till all the consumers are probed.
Quoted 5 lines
Answer: Let's say a consumer device Z has suppliers A, B and C. If the linking fails at A and you return immediately, then B and C could probe and then figure that they have no more consumers (they don't see a link to Z) and turn off their resources. And Z could fail catastrophically.
Then I think that this approach is fatally flawed in the current implementation.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
I'm waiting to hear how it is fatally flawed. But maybe this is just a misunderstanding of the problem?
Fatally flawed because it does not handle modules that add a consumer device when the module is loaded.
If you are talking about modules adding child devices of the device they are managing, then that's handled correctly later in the series.
They may or they may not. I do not know. I am not going to audit all current cases of devices being added to check that relationship and I am not going to monitor all future patches that add devices. Adding devices is an existing pattern of behavior that the new feature must be able to handle.
I have not looked at patch 6 yet (the place where modules adding child devices is handled). I am guessing that patch 6 could be made more general to remove the parent child relationship restriction.
Quoted 5 lines
If you are talking about modules adding devices that aren't defined in DT, then right, I'm not trying to handle that. The module needs to make sure it keeps the resources needed for new devices it's adding are in the right state or need to add the right device links.
I am not talking about devices that are not defined in the devicetree.
Quoted 16 lines
Quoted 16 lines
Quoted 4 lines
In the text below, I'm not sure if you mixing up two different things or just that your wording it a bit ambiguous. So pardon my nitpick to err on the side of clarity.
Please do nitpick. Clarity is good.
Quoted 5 lines
Quoted 2 lines
A device can be added by a module that is loaded.
No, in the example I gave,
of_platform_default_populate_init()would add all 3 of those devices duringarch_initcall_sync().The example you gave does not cover all use cases.
Quoted 8 lines
Quoted 2 lines
There are modules that add devices when the module is loaded. You can not ignore systems using such modules.
I'll have to agree to disagree on that. While I understand that the design should be good and I'm happy to work on that, you can't insist that a patch series shouldn't be allowed because it's only improving 99% of the cases and leaves the other 1% in the status quo. You are just going to bring the kernel development to a grinding halt.
No, you do not get to disagree on that. And you are presenting a straw man argument.
You are proposing a new feature that contributes fragility and complexity to the house of cards that device instantiation and driver probing already is.
The feature is clever but it is intertwined into an area that is already complex and in many cases difficult to work within.
I had hoped that the feature was robust enough and generic enough to accept. The proposed feature is a hack to paper over a specific problem that you are facing. I had hoped that the feature would appear generic enough that I would not have to regard it as an attempt to paper over the real problem. I have not given up this hope yet but I still am quite cautious about this approach to addressing your use case.
You have a real bug. I have told you how to fix the real bug. And you have ignored my suggestion. (To be honest, I do not know for sure that my suggestion is feasible, but on the surface it appears to be.) Again, my suggestion is to have the boot loader pass information to the kernel (via a chosen property) telling the kernel which devices the bootloader has enabled power to. The power subsystem would use that information early in boot to do a "get" on the power supplier (I am not using precise power subsystem terminology, but it should be obvious what I mean). The consumer device driver would also have to be aware of the information passed via the chosen property because the power subsystem has done the "get" on the consumer devices behalf (exactly how the consumer gets that information is an implementation detail). This approach is more direct, less subtle, less fragile.
Quoted 24 lines
Quoted 12 lines
Quoted 11 lines
Quoted 4 lines
In that case the device
was not present at late boot when the suppliers may turn off their resources.In that case, the _drivers_ for those devices aren't present at late boot. So that they can't request to keep the resources on for their consumer devices. Since there are no consumer requests on resources, the suppliers turn off their resources at late boot (since there isn't a better location as of today). The
sync_state()call back added in a subsequent patche in this series will provide the better location.Quoted 2 lines
And the
sync_state()call back will not deal with modules that add consumer devices when the module is loaded, correct?Depends. If it's just more devices from DT, then it'll be fine. If it's not, then the module needs to take care of the needs of devices it's adding.>
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
(I am assuming the details since I have not reviewed the patches later in the series that implement this part.)
Am I missing something?
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
I think you are mixing up devices getting added/populated with drivers getting loaded as modules?
Only some modules add devices when they are loaded. But these modules do exist.
Out of the billions of Android devices, how many do you see this happening in?
The Linux kernel is not just used by Android devices.
-Frank
Quoted 4 lines
Thanks,
Saravana
Re: [PATCH v7 3/7] of/platform: Add functional dependency link from DT bindings
Quoted 73 lines · Frank Rowand
Quoted 56 lines · Saravana Kannan
Quoted 55 lines · Frank Rowand
Quoted 53 lines · Saravana Kannan
Quoted 51 lines · Frank Rowand
Quoted 6 lines · Saravana Kannan
Quoted 5 lines · Frank Rowand
Quoted 3 lines · Saravana Kannan
Add device-links after the devices are created (but before they are probed) by looking at common DT bindings like clocks and interconnects.
< very big snip (lots of comments that deserve answers) >
Quoted 13 lines
Quoted 13 lines
/**
*of_link_property- TODO:
* dev:
*con_np:
* prop:
*
* TODO...
*
* Any failed attempt to create a link will NOT result in an immediate return.
*of_link_property()must create all possible links even when one of more
* attempts to create a link fail.Quoted 7 lines
Quoted 3 lines
Why? isn't one failure enough to prevent probing this device? Continuing to scan just results in extra work... which will be repeated every time
device_link_check_waiting_consumers()is calledContext:
As I said in the cover letter, avoiding unnecessary probes is just one of the reasons for this patch. The other (arguably more important)Agree that it is more important.
Quoted 7 lines
reason for this patch is to make sure suppliers know that they have consumers that are yet to be probed. That way, suppliers can leave their resource on AND in the right state if they were left on by the bootloader. For example, if a clock was left on and at 200 MHz, the clock provider needs to keep that clock ON and at 200 MHz till all the consumers are probed.
Quoted 5 lines
Answer: Let's say a consumer device Z has suppliers A, B and C. If the linking fails at A and you return immediately, then B and C could probe and then figure that they have no more consumers (they don't see a link to Z) and turn off their resources. And Z could fail catastrophically.
Then I think that this approach is fatally flawed in the current implementation.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
I'm waiting to hear how it is fatally flawed. But maybe this is just a misunderstanding of the problem?
Fatally flawed because it does not handle modules that add a consumer device when the module is loaded.
If you are talking about modules adding child devices of the device they are managing, then that's handled correctly later in the series.
They may or they may not. I do not know. I am not going to audit all current cases of devices being added to check that relationship and I am not going to monitor all future patches that add devices. Adding devices is an existing pattern of behavior that the new feature must be able to handle.
Quoted 3 lines
I have not looked at patch 6 yet (the place where modules adding child devices is handled). I am guessing that patch 6 could be made more general to remove the parent child relationship restriction.
Please do look into it then. I think it already handles all cases.
Quoted 7 lines
Quoted 5 lines
If you are talking about modules adding devices that aren't defined in DT, then right, I'm not trying to handle that. The module needs to make sure it keeps the resources needed for new devices it's adding are in the right state or need to add the right device links.
I am not talking about devices that are not defined in the devicetree.
In that case, I'm sure my patch series handle all the scenarios correctly. Here's why:
1. For all the top level devices the patches you've reviewed already show how it's handled correctly.
2. All other devices in the DT are by definition the child devices of the top level devices and patch 6 handles those cases.
Hopefully this shows to you that all DT cases are handled correctly.
Quoted 24 lines
Quoted 15 lines
Quoted 15 lines
Quoted 3 lines
In the text below, I'm not sure if you mixing up two different things or just that your wording it a bit ambiguous. So pardon my nitpick to err on the side of clarity.
Please do nitpick. Clarity is good.
Quoted 5 lines
Quoted 2 lines
A device can be added by a module that is loaded.
No, in the example I gave,
of_platform_default_populate_init()would add all 3 of those devices duringarch_initcall_sync().The example you gave does not cover all use cases.
Quoted 8 lines
Quoted 2 lines
There are modules that add devices when the module is loaded. You can not ignore systems using such modules.
I'll have to agree to disagree on that. While I understand that the design should be good and I'm happy to work on that, you can't insist that a patch series shouldn't be allowed because it's only improving 99% of the cases and leaves the other 1% in the status quo. You are just going to bring the kernel development to a grinding halt.
Quoted 6 lines
No, you do not get to disagree on that. And you are presenting a straw man argument.
You are proposing a new feature that contributes fragility and complexity to the house of cards that device instantiation and driver probing already is.
Any piece of code is going to "add complexity". It's a question of benefits vs complexity. Also, I'm mostly reusing existing device links API. The majority of the complexity is in parsing DT. The driver core maintainers seem to be fine with adding sync_state() support for device links (that is independent of DT).
Quoted 5 lines
The feature is clever but it is intertwined into an area that is already complex and in many cases difficult to work within.
I had hoped that the feature was robust enough and generic enough to accept.
What I'm doing IS the most generic solution instead of doing the same work multiple times at a framework level. Also, for multi-function devices, framework level solutions would be worse.
Quoted 9 lines
The proposed feature is a hack to paper over a specific problem that you are facing. I had hoped that the feature would appear generic enough that I would not have to regard it as an attempt to paper over the real problem. I have not given up this hope yet but I still am quite cautious about this approach to addressing your use case.
You have a real bug. I have told you how to fix the real bug. And you have ignored my suggestion. (To be honest, I do not know for sure that my suggestion is feasible, but on the surface it appears to be.)
I'd actually say that your proposal is what's trying to paper over a generic problem by saying it's specific to one or a few set of resources. And it looks feasible to you because you haven't dove deep into this issue.
Quoted 11 lines
Again,
my suggestion is to have the boot loader pass information to the kernel (via a chosen property) telling the kernel which devices the bootloader has enabled power to. The power subsystem would use that information early in boot to do a "get" on the power supplier (I am not using precise power subsystem terminology, but it should be obvious what I mean). The consumer device driver would also have to be aware of the information passed via the chosen property because the power subsystem has done the "get" on the consumer devices behalf (exactly how the consumer gets that information is an implementation detail). This approach is more direct, less subtle, less fragile.
I'll have to disagree on your claim. You are adding unnecessary bootloader dependency when the kernel is completely capable of handling this on its own. You are requiring explicit "gets" by suppliers and then hoping all the consumers do the corresponding "puts" to balance it out. Somehow the consumers need to know which suppliers have parsed which bootloader input. And it's barely scratching the surface of the problem.
You are assuming this has to do with just power when it can be clocks, interconnects, etc. Why solve this repeated for each framework when you can have a generic solution?
Also, while I understand what you mean by "get" it's not going to be as simple as a reference count to keep the resource on. In reality you'll need more complex handling. For example, having to keep a voltage rail at or above X mV because one consumer might fail if the voltage is < X mV. Or making sure a clock never goes about the bootloader set frequency before all the consumer drivers are probed to avoid overclocking one of the consumers. Trying to have this explicitly coordinated across multiple drivers would be a nightmare. It gets even more complicated with interconnects.
With my patch series, the consumers don't need to do anything. They just probe as usual. The suppliers don't need to track or coordinate with any consumers. For example, regulator suppliers just need to keep the voltage rail at (or above) the level that the boot loader left it on at and then apply the aggregated requests from their APIs once they get the sync_state() callback. And it actually works -- tested for regulators and clocks (and maybe even interconnects -- I forgot) in a device I have.
Quoted 26 lines
Quoted 26 lines
Quoted 11 lines
Quoted 10 lines
Quoted 3 lines
In that case the device
was not present at late boot when the suppliers may turn off their resources.In that case, the _drivers_ for those devices aren't present at late boot. So that they can't request to keep the resources on for their consumer devices. Since there are no consumer requests on resources, the suppliers turn off their resources at late boot (since there isn't a better location as of today). The
sync_state()call back added in a subsequent patche in this series will provide the better location.Quoted 2 lines
And the
sync_state()call back will not deal with modules that add consumer devices when the module is loaded, correct?Depends. If it's just more devices from DT, then it'll be fine. If it's not, then the module needs to take care of the needs of devices it's adding.>
Quoted 9 lines
Quoted 8 lines
Quoted 5 lines
(I am assuming the details since I have not reviewed the patches later in the series that implement this part.)
Am I missing something?
I think you are mixing up devices getting added/populated with drivers getting loaded as modules?
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
Only some modules add devices when they are loaded. But these modules do exist.
Out of the billions of Android devices, how many do you see this happening in?
The Linux kernel is not just used by Android devices.
Ofcourse Linux is used by more than just Android. And Android is just an ARM64(32) distribution. But how many platforms do you have where a module adds devices that are not part of DT (because I'm handling the DT part fine -- see other emails)? How does that count compare to millions of products that can use this feature? And I'm not breaking any of the existing platforms that don't use DT either. So saying I have to fix this for 100% of the use cases for Linux before I can remove the roadblocks for a common ARM64 kernel that can run on any ARM64 platform seems like an unreasonable position.
-Saravana
Re: [PATCH v7 3/7] of/platform: Add functional dependency link from DT bindings
Quoted 2 lines · Saravana Kannan
Quoted 1 line · Frank Rowand
< snip - the stuff I snipped deserves reply, but I want to focus on just
one topic for this reply >
Quoted 8 lines
Quoted 3 lines
You have a real bug. I have told you how to fix the real bug. And you have ignored my suggestion. (To be honest, I do not know for sure that my suggestion is feasible, but on the surface it appears to be.)
I'd actually say that your proposal is what's trying to paper over a generic problem by saying it's specific to one or a few set of resources. And it looks feasible to you because you haven't dove deep into this issue.
Not saying it is specific to one or a few sets of resources. The proposal suggests handling every single consumer supplier relationship for which the bootloader has enabled a supplier resource via an explicit message communicating the enabled resources. And directly handling those exact resources.
Think about the definition of "paper over" vs "directly address".
Quoted 13 lines
Quoted 12 lines
Again,
my suggestion is to have the boot loader pass information to the kernel (via a chosen property) telling the kernel which devices the bootloader has enabled power to. The power subsystem would use that information early in boot to do a "get" on the power supplier (I am not using precise power subsystem terminology, but it should be obvious what I mean). The consumer device driver would also have to be aware of the information passed via the chosen property because the power subsystem has done the "get" on the consumer devices behalf (exactly how the consumer gets that information is an implementation detail). This approach is more direct, less subtle, less fragile.
Quoted 7 lines
I'll have to disagree on your claim. You are adding unnecessary bootloader dependency when the kernel is completely capable of handling this on its own. You are requiring explicit "gets" by suppliers and then hoping all the consumers do the corresponding "puts" to balance it out. Somehow the consumers need to know which suppliers have parsed which bootloader input. And it's barely scratching the surface of the problem.
OK, let me flesh out a possible implementation just a little bit.
This is focused on devicetree, as is your patch series. For ACPI a parallel implementation would exist.
The bootloader chosen property could be a list of tuples, each tuple
containing: consumer phandle, supplier phandle. Each tuple could contain more data if the implementation demands, but I'm trying to keep it simple to illustrate the concept.
In early-ish boot a core function processes the chosen property. For each consumer / supplier pair, the supplier compatible could be used to determine the supplier type. (This might not be enough info to determine the supplier type - maybe the consumer property that points to the supplier will also have to be specified in the chosen property tuple, or maybe a supplier type could be added to the tuple.) Given the consumer, supplier, and resource type the appropriate "get" would be done.
Late in boot, and possible repeated after modules are loaded, a core function would scan the chosen property tuples, and for each consumer / supplier pair, if both the consumer and the supplier drivers are bound, it would be ASSUMED that it is ok to do the appropriate type of "put", and the "put" would be done.
Quoted 4 lines
You are assuming this has to do with just power when it can be clocks, interconnects, etc. Why solve this repeated for each framework when you can have a generic solution?
No such assumption.
Quoted 11 lines
Also, while I understand what you mean by "get" it's not going to be as simple as a reference count to keep the resource on. In reality you'll need more complex handling. For example, having to keep a voltage rail at or above X mV because one consumer might fail if the voltage is < X mV. Or making sure a clock never goes about the bootloader set frequency before all the consumer drivers are probed to avoid overclocking one of the consumers. Trying to have this explicitly coordinated across multiple drivers would be a nightmare. It gets even more complicated with interconnects.
Quoted 9 lines
With my patch series, the consumers don't need to do anything. They just probe as usual. The suppliers don't need to track or coordinate with any consumers. For example, regulator suppliers just need to keep the voltage rail at (or above) the level that the boot loader left it on at and then apply the aggregated requests from their APIs once they get the
sync_state()callback. And it actually works -- tested for regulators and clocks (and maybe even interconnects -- I forgot) in a device I have.
And same for the possible implementation I sketched above. The equivalent of the sync_state() callback would be done by the end of boot (potentially repeated after each module loads) core function making a similar call. Hand waving here about what suppliers to call.
Of course this is not the only way to implement my concept, just an example to suggest that it might be feasible and it might work.
< snip >
-Frank
PATCH v7 4/7 driver core: Add sync_state driver/bus callback
In reply to Saravana KannanThis sync_state driver/bus callback is called once all the consumers of a supplier have probed successfully.
This allows the supplier device's driver/bus to sync the supplier device's state to the software state with the guarantee that all the consumers are actively managing the resources provided by the supplier device.
To maintain backwards compatibility and ease transition from existing frameworks and resource cleanup schemes, late_initcall_sync is the earliest when the sync_state callback might be called.
There is no upper bound on the time by which the sync_state callback has to be called. This is because if a consumer device never probes, the supplier has to maintain its resources in the state left by the bootloader. For example, if the bootloader leaves the display backlight at a fixed voltage and the backlight driver is never probed, you don't want the backlight to ever be turned off after boot up.
Also, when multiple devices are added after kernel init, some suppliers could be added before their consumer devices get added. In these instances, the supplier devices could get their sync_state callback called right after they probe because the consumers devices haven't had a chance to create device links to the suppliers.
To handle this correctly, this change also provides APIs to pause/resume sync state callbacks so that when multiple devices are added, their sync_state callback evaluation can be postponed to happen after all of them are added.
--- drivers/base/core.c | 65 ++++++++++++++++++++++++++++++++++++++++++ include/linux/device.h | 23 +++++++++++++++ 2 files changed, 88 insertions(+)
drivers/base/core.c +65 -0
@@ -46,6 +46,8 @@ early_param("sysfs.deprecated", sysfs_deprecated_setup);/* Device links support. */ static LIST_HEAD (wait_for_suppliers);static DEFINE_MUTEX (wfs_lock); +static LIST_HEAD (deferred_sync); +static unsigned int supplier_sync_state_disabled;#ifdef CONFIG_SRCU static DEFINE_MUTEX (device_links_lock);
@@ -634,6 +636,62 @@ int device_links_check_suppliers(struct device *dev)return ret; } +static void __device_links_supplier_sync_state (struct device *dev) +{ +struct device_link *link; + +if (dev->state_synced) +return ; + +list_for_each_entry (link, &dev->links.consumers, s_node) { +if (link->flags &DL_FLAG_STATELESS ) +continue ; +if (link->status !=DL_STATE_ACTIVE ) +return ; + } + +if (dev->bus->sync_state) + dev->bus->sync_state (dev); +else if (dev->driver && dev->driver->sync_state) + dev->driver->sync_state (dev); + + dev->state_synced = true; +} + +void device_links_supplier_sync_state_pause (void ) +{ +device_links_write_lock (); + supplier_sync_state_disabled++; +device_links_write_unlock (); +} + +void device_links_supplier_sync_state_resume (void ) +{ +struct device *dev, *tmp; + +device_links_write_lock (); +if (!supplier_sync_state_disabled) { +WARN (true,"Unmatched sync_state pause/resume!" ); + goto out; + } + supplier_sync_state_disabled--; +if (supplier_sync_state_disabled) + goto out; + +list_for_each_entry_safe (dev, tmp, &deferred_sync, links.defer_sync) { +__device_links_supplier_sync_state (dev); +list_del_init (&dev->links.defer_sync); + } +out: +device_links_write_unlock (); +} + +static void __device_links_supplier_defer_sync (struct device *sup) +{ +if (list_empty (&sup->links.defer_sync)) +list_add_tail (&sup->links.defer_sync, &deferred_sync); +} +/** * device_links_driver_bound - Update device links after probing its driver. * @dev: Device to update the links for.
@@ -678,6 +736,11 @@ void device_links_driver_bound(struct device *dev)WARN_ON (link->status !=DL_STATE_CONSUMER_PROBE );WRITE_ONCE (link->status,DL_STATE_ACTIVE ); + +if (supplier_sync_state_disabled) +__device_links_supplier_defer_sync (link->supplier); +else +__device_links_supplier_sync_state (link->supplier); } dev->links.status =DL_DEV_DRIVER_BOUND ;
@@ -787,6 +850,7 @@ void device_links_driver_cleanup(struct device *dev)WRITE_ONCE (link->status,DL_STATE_DORMANT ); } +list_del_init (&dev->links.defer_sync);__device_links_no_driver (dev);device_links_write_unlock ();
@@ -1758,6 +1822,7 @@ void device_initialize(struct device *dev)INIT_LIST_HEAD (&dev->links.consumers);INIT_LIST_HEAD (&dev->links.suppliers);INIT_LIST_HEAD (&dev->links.needs_suppliers); +INIT_LIST_HEAD (&dev->links.defer_sync); dev->links.status =DL_DEV_NO_DRIVER ; }EXPORT_SYMBOL_GPL (device_initialize);
include/linux/device.h +23 -0
@@ -84,6 +84,8 @@ extern void bus_remove_file(struct bus_type *, struct bus_attribute *);* available at the time this function is called. As in, the * function should NOT stop at the first failed device link if * other unlinked supplier devices are present in the system. +* This is necessary for the sync_state() callback to work +* correctly. * * Return 0 if device links have been successfully created to all * the suppliers of this device. Return an error if some of the
@@ -91,6 +93,13 @@ extern void bus_remove_file(struct bus_type *, struct bus_attribute *);* reattempted in the future. * @probe: Called when a new device or driver add to this bus, and callback * the specific driver's probe to initial the matched device. +* @sync_state: Called to sync device state to software state after all the +* state tracking consumers linked to this device (present at +* the time of late_initcall) have successfully bound to a +* driver. If the device has no consumers, this function will +* be called at late_initcall_sync level. If the device has +* consumers that are never bound to a driver, this function +* will never get called until they do. * @remove: Called when a device removed from this bus. * @shutdown: Called at shut-down time to quiesce the device. *
@@ -135,6 +144,7 @@ struct bus_type {int (*uevent)(struct device *dev,struct kobj_uevent_env *env);int (*add_links)(struct device *dev);int (*probe)(struct device *dev); +void (*sync_state)(struct device *dev);int (*remove)(struct device *dev);void (*shutdown)(struct device *dev);
@@ -280,6 +290,13 @@ enum probe_type {* @probe: Called to query the existence of a specific device, * whether this driver can work with it, and bind the driver * to a specific device. +* @sync_state: Called to sync device state to software state after all the +* state tracking consumers linked to this device (present at +* the time of late_initcall) have successfully bound to a +* driver. If the device has no consumers, this function will +* be called at late_initcall_sync level. If the device has +* consumers that are never bound to a driver, this function +* will never get called until they do. * @remove: Called when the device is removed from the system to * unbind a device from this driver. * @shutdown: Called at shut-down time to quiesce the device.
@@ -318,6 +335,7 @@ struct device_driver {int (*edit_links)(struct device *dev);int (*probe) (struct device *dev); +void (*sync_state)(struct device *dev);int (*remove) (struct device *dev);void (*shutdown) (struct device *dev);int (*suspend) (struct device *dev,pm_message_t state);
@@ -921,12 +939,14 @@ enum dl_dev_state {* @suppliers: List of links to supplier devices. * @consumers: List of links to consumer devices. * @needs_suppliers: Hook to global list of devices waiting for suppliers. +* @defer_sync: Hook to global list of devices that have deferred sync_state. * @status: Driver status information. */ struct dev_links_info {struct list_head suppliers;struct list_head consumers;struct list_head needs_suppliers; +struct list_head defer_sync;enum dl_dev_state status; };
@@ -1094,6 +1114,7 @@ struct device { bool offline:1; bool of_node_reused:1; bool has_edit_links:1; + bool state_synced:1;#if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_DEVICE ) || \ defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU ) || \ defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU_ALL )
@@ -1437,6 +1458,8 @@ struct device_link *device_link_add(struct device *consumer,void device_link_del (struct device_link *link);void device_link_remove (void *consumer,struct device *supplier);void device_link_remove_from_wfs (struct device *consumer); +void device_links_supplier_sync_state_pause (void ); +void device_links_supplier_sync_state_resume (void );#ifndef dev_fmt#define dev_fmt (fmt) fmt
Signature · 1 line
2.22.0.709.g102302147b-goog
PATCH v7 5/7 of/platform: Pause/resume sync state during init and of_platform_populate()
In reply to Saravana KannanWhen all the top level devices are populated from DT during kernel init, the supplier devices could be added and probed before the consumer devices are added and linked to the suppliers. To avoid the sync_state() callback from being called prematurely, pause the sync_state() callbacks before populating the devices and resume them at late_initcall_sync().
Similarly, when children devices are populated after kernel init using of_platform_populate(), there could be supplier-consumer dependencies between the children devices that are populated. To avoid the same problem with sync_state() being called prematurely, pause and resume sync_state() callbacks across of_platform_populate().
--- drivers/of/platform.c | 11 +++++++++++ 1 file changed, 11 insertions(+)
drivers/of/platform.c +11 -0
@@ -485,6 +485,7 @@ int of_platform_populate(struct device_node *root,pr_debug ("%s()\n" , __func__);pr_debug (" starting at: %pOF\n" , root); +device_links_supplier_sync_state_pause ();for_each_child_of_node (root, child) { rc =of_platform_bus_create (child, matches, lookup, parent, true);if (rc) {
@@ -492,6 +493,8 @@ int of_platform_populate(struct device_node *root,break ; } } +device_links_supplier_sync_state_resume (); +of_node_set_flag (root,OF_POPULATED_BUS );of_node_put (root);
@@ -688,6 +691,7 @@ static int __init of_platform_default_populate_init(void)return -ENODEV ; platform_bus_type.add_links = of_link_to_suppliers; +device_links_supplier_sync_state_pause ();/* * Handle certain compatibles explicitly, since we don't want to create * platform_devices for every node in /reserved-memory with a
@@ -708,6 +712,13 @@ static int __init of_platform_default_populate_init(void) return 0; }arch_initcall_sync (of_platform_default_populate_init); + +static int __init of_platform_sync_state_init (void ) +{ +device_links_supplier_sync_state_resume (); +return 0 ; +} +late_initcall_sync (of_platform_sync_state_init);#endif int of_platform_device_destroy (struct device *dev,void *data)
Signature · 1 line
2.22.0.709.g102302147b-goog
PATCH v7 6/7 of/platform: Create device links for all child-supplier depencencies
In reply to Saravana KannanA parent device can have child devices that it adds when it probes. But this probing of the parent device can happen way after kernel init is done -- for example, when the parent device's driver is loaded as a module.
In such cases, if the child devices depend on a supplier in the system, we need to make sure the supplier gets the sync_state() callback only after these child devices are added and probed.
To achieve this, when creating device links for a device by looking at its DT node, don't just look at DT references at the top node level. Look at DT references in all the descendant nodes too and create device links from the ancestor device to all these supplier devices.
This way, when the parent device probes and adds child devices, the child devices can then create their own device links to the suppliers and further delay the supplier's sync_state() callback to after the child devices are probed.
Example:
In this illustration, -> denotes DT references and indentation represents child status.
Device node A
Device node B -> D Device node C -> B, D
Device node D
Assume all these devices have their drivers loaded as modules.
Without this patch, this is the sequence of events:
1. D is added.
2. A is added.
3. Device D probes.
4. Device D gets its sync_state() callback.
5. Device B and C might malfunction because their resources got altered/turned off before they can make active requests for them.
With this patch, this is the sequence of events:
1. D is added.
2. A is added and creates device links to D.
3. Device link from A to B is not added because A is a parent of B.
4. Device D probes.
5. Device D does not get it's sync_state() callback because consumer A hasn't probed yet.
5. Device A probes.
5. a. Devices B and C are added.
5. b. Device links from B and C to D are added.
5. c. Device A's probe completes.
6. Device D does not get it's sync_state() callback because consumer A has probed but consumers B and C haven't probed yet.
7. Device B and C probe.
8. Device D gets it's sync_state() callback because all its consumers have probed.
9. None of the devices malfunction.
--- drivers/of/platform.c | 27 +++++++++++++++++++-------- 1 file changed, 19 insertions(+), 8 deletions(-)
drivers/of/platform.c +19 -8
@@ -655,24 +655,35 @@ static bool of_link_property(struct device *dev, struct device_node *con_np,return done ?0 : -ENODEV ; } +static int __of_link_to_suppliers (struct device *dev, +struct device_node *con_np) +{ +struct device_node *child; +struct property *p; +bool done = true; + +for_each_property_of_node (con_np, p) +if (of_link_property (dev, con_np, p->name)) + done = false; + +for_each_child_of_node (con_np, child) +if (__of_link_to_suppliers (dev, child)) + done = false; + +return done ?0 : -ENODEV ; +} +static bool of_devlink;core_param (of_devlink, of_devlink, bool,0 );static int of_link_to_suppliers (struct device *dev) { -struct property *p; -bool done = true; -if (!of_devlink)return 0 ;if (unlikely (!dev->of_node))return 0 ; -for_each_property_of_node (dev->of_node, p) -if (of_link_property (dev, dev->of_node, p->name)) - done = false; - -return done ?0 : -ENODEV ; +return __of_link_to_suppliers (dev, dev->of_node); }#ifndef CONFIG_PPC
Signature · 1 line
2.22.0.709.g102302147b-goog
PATCH v7 7/7 of/platform: Don't create device links for default busses
In reply to Saravana KannanDefault busses also have devices created for them. But there's no point in creating device links for them. It's especially wasteful as it'll cause the traversal of the entire device tree and also spend a lot of time checking and figuring out that creating those links isn't allowed. So check for default busses and skip trying to create device links for them.
--- drivers/of/platform.c | 2 ++ 1 file changed, 2 insertions(+)
drivers/of/platform.c +2 -0
@@ -682,6 +682,8 @@ static int of_link_to_suppliers(struct device *dev)return 0 ;if (unlikely (!dev->of_node))return 0 ; +if (of_match_node (of_default_bus_match_table, dev->of_node)) +return 0 ;return __of_link_to_suppliers (dev, dev->of_node); }
Signature · 1 line
2.22.0.709.g102302147b-goog
Greg Kroah-Hartman
maintainerQuoted 84 lines · Saravana Kannan
Add device-links to track functional dependencies between devices after they are created (but before they are probed) by looking at their common DT bindings like clocks, interconnects, etc.
Having functional dependencies automatically added before the devices are probed, provides the following benefits:
- Optimizes device probe order and avoids the useless work of attempting probes of devices that will not probe successfully (because their suppliers aren't present or haven't probed yet).
For example, in a commonly available mobile SoC, registering just one consumer device's driver at an initcall level earlier than the supplier device's driver causes 11 failed probe attempts before the consumer device probes successfully. This was with a kernel with all the drivers statically compiled in. This problem gets a lot worse if all the drivers are loaded as modules without direct symbol dependencies.
- Supplier devices like clock providers, interconnect providers, etc need to keep the resources they provide active and at a particular state(s) during boot up even if their current set of consumers don't request the resource to be active. This is because the rest of the consumers might not have probed yet and turning off the resource before all the consumers have probed could lead to a hang or undesired user experience.
Some frameworks (Eg: regulator) handle this today by turning off "unused" resources at
late_initcall_syncand hoping all the devices have probed by then. This is not a valid assumption for systems with loadable modules. Other frameworks (Eg: clock) just don't handle this due to the lack of a clear signal for when they can turn off resources. This leads to downstream hacks to handle cases like this that can easily be solved in the upstream kernel.By linking devices before they are probed, we give suppliers a clear count of the number of dependent consumers. Once all of the consumers are active, the suppliers can turn off the unused resources without making assumptions about the number of consumers.
By default we just add device-links to track "driver presence" (probe succeeded) of the supplier device. If any other functionality provided by device-links are needed, it is left to the consumer/supplier devices to change the link when they probe.
v1 -> v2:
- Drop patch to speed upof_find_device_by_node()
- Drop depends-on property and use existing bindingsv2 -> v3:
- Refactor the code to have driver core initiate the linking of devs
- Have driver core link consumers to supplier before it's probed
- Add support for drivers to edit the device links before probingv3 -> v4:
- Testededit_links()on system with cyclic dependency. Works.
- Added some checks to make sure device link isn't attempted from parent device node to child device node.
- Added way to pause/resumesync_statecallbacks acrossof_platform_populate().
- Recursively parse DT node to create device links from parent to suppliers of parent and all child nodes.v4 -> v5:
- Fixed copy-pasta bugs with linked list handling
- Walk up the phandle reference till I find an actual device (needed for regulators to work)
- Added support for linking devices from regulator DT bindings
- Tested the whole series again to make sure cyclic dependencies are broken withedit_links()and regulator links are created properly.v5 -> v6:
- Split, squashed and reordered some of the patches.
- Refactored the device linking code to follow the same code pattern for any property.v6 -> v7:
- No functional changes.
- Renamed i to index
- Added comment to clarify not having to check property name for every index
- Added "matched" variable to clarify code. No functional change.
- Added comments toinclude/linux/device.hforadd_links()
Quoted 4 lines
I've also not updated this patch series to handle the new patch [1] from Rafael. Will do that once this patch series is close to being Acked.
[1] - https://lore.kernel.org/lkml/3121545.4lOhFoIcdQ@kreacher/
This looks sane to me. Anyone have any objections for me queueing this up for my tree to get into linux-next now?
thanks,
greg k-h
Quoted 85 lines · Greg Kroah-Hartman
Quoted 84 lines · Saravana Kannan
Add device-links to track functional dependencies between devices after they are created (but before they are probed) by looking at their common DT bindings like clocks, interconnects, etc.
Having functional dependencies automatically added before the devices are probed, provides the following benefits:
- Optimizes device probe order and avoids the useless work of attempting probes of devices that will not probe successfully (because their suppliers aren't present or haven't probed yet).
For example, in a commonly available mobile SoC, registering just one consumer device's driver at an initcall level earlier than the supplier device's driver causes 11 failed probe attempts before the consumer device probes successfully. This was with a kernel with all the drivers statically compiled in. This problem gets a lot worse if all the drivers are loaded as modules without direct symbol dependencies.
- Supplier devices like clock providers, interconnect providers, etc need to keep the resources they provide active and at a particular state(s) during boot up even if their current set of consumers don't request the resource to be active. This is because the rest of the consumers might not have probed yet and turning off the resource before all the consumers have probed could lead to a hang or undesired user experience.
Some frameworks (Eg: regulator) handle this today by turning off "unused" resources at
late_initcall_syncand hoping all the devices have probed by then. This is not a valid assumption for systems with loadable modules. Other frameworks (Eg: clock) just don't handle this due to the lack of a clear signal for when they can turn off resources. This leads to downstream hacks to handle cases like this that can easily be solved in the upstream kernel.By linking devices before they are probed, we give suppliers a clear count of the number of dependent consumers. Once all of the consumers are active, the suppliers can turn off the unused resources without making assumptions about the number of consumers.
By default we just add device-links to track "driver presence" (probe succeeded) of the supplier device. If any other functionality provided by device-links are needed, it is left to the consumer/supplier devices to change the link when they probe.
v1 -> v2:
- Drop patch to speed upof_find_device_by_node()
- Drop depends-on property and use existing bindingsv2 -> v3:
- Refactor the code to have driver core initiate the linking of devs
- Have driver core link consumers to supplier before it's probed
- Add support for drivers to edit the device links before probingv3 -> v4:
- Testededit_links()on system with cyclic dependency. Works.
- Added some checks to make sure device link isn't attempted from parent device node to child device node.
- Added way to pause/resumesync_statecallbacks acrossof_platform_populate().
- Recursively parse DT node to create device links from parent to suppliers of parent and all child nodes.v4 -> v5:
- Fixed copy-pasta bugs with linked list handling
- Walk up the phandle reference till I find an actual device (needed for regulators to work)
- Added support for linking devices from regulator DT bindings
- Tested the whole series again to make sure cyclic dependencies are broken withedit_links()and regulator links are created properly.v5 -> v6:
- Split, squashed and reordered some of the patches.
- Refactored the device linking code to follow the same code pattern for any property.v6 -> v7:
- No functional changes.
- Renamed i to index
- Added comment to clarify not having to check property name for every index
- Added "matched" variable to clarify code. No functional change.
- Added comments toinclude/linux/device.hforadd_links()
Quoted 8 lines
Quoted 4 lines
I've also not updated this patch series to handle the new patch [1] from Rafael. Will do that once this patch series is close to being Acked.
[1] - https://lore.kernel.org/lkml/3121545.4lOhFoIcdQ@kreacher/
This looks sane to me. Anyone have any objections for me queueing this up for my tree to get into linux-next now?
I would like for the series to get into linux-next sooner than later, and spend some time there.
I am _slightly_ more optimistic than Rob that sitting in linux-next for an extended period might reveal any latent issues, so I would like for the series to be in linux-next for an extended period of time. (Yes, my understanding is that Linus does not like patches to be in linux-next if they are not targeted for the next merge window, but I prefer that this patch series spend as much time in linux-next as possible).
I have been waiting for the changes to settle down before bringing up the issue of devicetree overlays. Now that the code seems to be settling down, I need to look at how these changes impact overlays. So I do not think the patches will be ready for a Linus pull request until overlays are considered.
-Frank
Quoted 5 lines
thanks,
greg k-h
Greg Kroah-Hartman
maintainerQuoted 89 lines · Frank Rowand
Quoted 88 lines · Greg Kroah-Hartman
Quoted 87 lines · Saravana Kannan
Add device-links to track functional dependencies between devices after they are created (but before they are probed) by looking at their common DT bindings like clocks, interconnects, etc.
Having functional dependencies automatically added before the devices are probed, provides the following benefits:
- Optimizes device probe order and avoids the useless work of attempting probes of devices that will not probe successfully (because their suppliers aren't present or haven't probed yet).
For example, in a commonly available mobile SoC, registering just one consumer device's driver at an initcall level earlier than the supplier device's driver causes 11 failed probe attempts before the consumer device probes successfully. This was with a kernel with all the drivers statically compiled in. This problem gets a lot worse if all the drivers are loaded as modules without direct symbol dependencies.
- Supplier devices like clock providers, interconnect providers, etc need to keep the resources they provide active and at a particular state(s) during boot up even if their current set of consumers don't request the resource to be active. This is because the rest of the consumers might not have probed yet and turning off the resource before all the consumers have probed could lead to a hang or undesired user experience.
Some frameworks (Eg: regulator) handle this today by turning off "unused" resources at
late_initcall_syncand hoping all the devices have probed by then. This is not a valid assumption for systems with loadable modules. Other frameworks (Eg: clock) just don't handle this due to the lack of a clear signal for when they can turn off resources. This leads to downstream hacks to handle cases like this that can easily be solved in the upstream kernel.By linking devices before they are probed, we give suppliers a clear count of the number of dependent consumers. Once all of the consumers are active, the suppliers can turn off the unused resources without making assumptions about the number of consumers.
By default we just add device-links to track "driver presence" (probe succeeded) of the supplier device. If any other functionality provided by device-links are needed, it is left to the consumer/supplier devices to change the link when they probe.
v1 -> v2:
- Drop patch to speed upof_find_device_by_node()
- Drop depends-on property and use existing bindingsv2 -> v3:
- Refactor the code to have driver core initiate the linking of devs
- Have driver core link consumers to supplier before it's probed
- Add support for drivers to edit the device links before probingv3 -> v4:
- Testededit_links()on system with cyclic dependency. Works.
- Added some checks to make sure device link isn't attempted from parent device node to child device node.
- Added way to pause/resumesync_statecallbacks acrossof_platform_populate().
- Recursively parse DT node to create device links from parent to suppliers of parent and all child nodes.v4 -> v5:
- Fixed copy-pasta bugs with linked list handling
- Walk up the phandle reference till I find an actual device (needed for regulators to work)
- Added support for linking devices from regulator DT bindings
- Tested the whole series again to make sure cyclic dependencies are broken withedit_links()and regulator links are created properly.v5 -> v6:
- Split, squashed and reordered some of the patches.
- Refactored the device linking code to follow the same code pattern for any property.v6 -> v7:
- No functional changes.
- Renamed i to index
- Added comment to clarify not having to check property name for every index
- Added "matched" variable to clarify code. No functional change.
- Added comments toinclude/linux/device.hforadd_links()I've also not updated this patch series to handle the new patch [1] from Rafael. Will do that once this patch series is close to being Acked.
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
[1] - https://lore.kernel.org/lkml/3121545.4lOhFoIcdQ@kreacher/
This looks sane to me. Anyone have any objections for me queueing this up for my tree to get into linux-next now?
I would like for the series to get into linux-next sooner than later, and spend some time there.
Ok, care to give me an ack for it? :)
thanks,
greg k-h
Hi Greg, Rob,
Quoted 93 lines · Greg Kroah-Hartman
Quoted 92 lines · Frank Rowand
Quoted 91 lines · Greg Kroah-Hartman
Quoted 88 lines · Saravana Kannan
Add device-links to track functional dependencies between devices after they are created (but before they are probed) by looking at their common DT bindings like clocks, interconnects, etc.
Having functional dependencies automatically added before the devices are probed, provides the following benefits:
- Optimizes device probe order and avoids the useless work of attempting probes of devices that will not probe successfully (because their suppliers aren't present or haven't probed yet).
For example, in a commonly available mobile SoC, registering just one consumer device's driver at an initcall level earlier than the supplier device's driver causes 11 failed probe attempts before the consumer device probes successfully. This was with a kernel with all the drivers statically compiled in. This problem gets a lot worse if all the drivers are loaded as modules without direct symbol dependencies.
- Supplier devices like clock providers, interconnect providers, etc need to keep the resources they provide active and at a particular state(s) during boot up even if their current set of consumers don't request the resource to be active. This is because the rest of the consumers might not have probed yet and turning off the resource before all the consumers have probed could lead to a hang or undesired user experience.
Some frameworks (Eg: regulator) handle this today by turning off "unused" resources at
late_initcall_syncand hoping all the devices have probed by then. This is not a valid assumption for systems with loadable modules. Other frameworks (Eg: clock) just don't handle this due to the lack of a clear signal for when they can turn off resources. This leads to downstream hacks to handle cases like this that can easily be solved in the upstream kernel.By linking devices before they are probed, we give suppliers a clear count of the number of dependent consumers. Once all of the consumers are active, the suppliers can turn off the unused resources without making assumptions about the number of consumers.
By default we just add device-links to track "driver presence" (probe succeeded) of the supplier device. If any other functionality provided by device-links are needed, it is left to the consumer/supplier devices to change the link when they probe.
v1 -> v2:
- Drop patch to speed upof_find_device_by_node()
- Drop depends-on property and use existing bindingsv2 -> v3:
- Refactor the code to have driver core initiate the linking of devs
- Have driver core link consumers to supplier before it's probed
- Add support for drivers to edit the device links before probingv3 -> v4:
- Testededit_links()on system with cyclic dependency. Works.
- Added some checks to make sure device link isn't attempted from parent device node to child device node.
- Added way to pause/resumesync_statecallbacks acrossof_platform_populate().
- Recursively parse DT node to create device links from parent to suppliers of parent and all child nodes.v4 -> v5:
- Fixed copy-pasta bugs with linked list handling
- Walk up the phandle reference till I find an actual device (needed for regulators to work)
- Added support for linking devices from regulator DT bindings
- Tested the whole series again to make sure cyclic dependencies are broken withedit_links()and regulator links are created properly.v5 -> v6:
- Split, squashed and reordered some of the patches.
- Refactored the device linking code to follow the same code pattern for any property.v6 -> v7:
- No functional changes.
- Renamed i to index
- Added comment to clarify not having to check property name for every index
- Added "matched" variable to clarify code. No functional change.
- Added comments toinclude/linux/device.hforadd_links()I've also not updated this patch series to handle the new patch [1] from Rafael. Will do that once this patch series is close to being Acked.
[1] - https://lore.kernel.org/lkml/3121545.4lOhFoIcdQ@kreacher/
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
This looks sane to me. Anyone have any objections for me queueing this up for my tree to get into linux-next now?
I would like for the series to get into linux-next sooner than later, and spend some time there.
Ok, care to give me an ack for it? :)
Rob opined to me that if you apply the series, it will go into 5.4 unless reverted. That is also what I would expect.
I'm going through the series carefully now. This is currently my highest priority task. I don't yet know if my comments will be minor, or whether I will have larger changes to request.
So I am not ready to ack the series yet.
-Frank
Quoted 5 lines
thanks,
greg k-h
Hi Saravana,
Quoted 4 lines · Saravana Kannan
Add device-links to track functional dependencies between devices after they are created (but before they are probed) by looking at their common DT bindings like clocks, interconnects, etc.
< snip >
I know that this series has moved on to versions 8 and 9. And some additional patches submitted.
Version 8 was a rebase to handle device_link changes. The version 8 patch 0/7 description of the changes did not note any functional changes, so I am assuming that my comments on version 7 are still applicable.
The version 9 changes do not impact my comments.
I am sending review comments on patches 1, 2, and 3. I will continue review of patches later in the series when the fall out from these review comments result in a new series.
-Frank