Add NUMA aware async_schedule calls
Subsystems (5)
v4 cover Add NUMA aware async_schedule calls
This patch set provides functionality that will help to improve the locality of the async_schedule calls used to provide deferred initialization.
This patch set originally started out with me focused on just the one call to async_schedule_domain in the nvdimm tree that was being used to defer the device_add call however after doing some digging I realized the scope of this was much broader than I had originally planned. As such I went through and reworked the underlying infrastructure down to replacing the queue_work call itself with a function of my own and opted to try and provide a NUMA aware solution that would work for a broader audience.
RFC->v1:
Dropped nvdimm patch to submit later
It relies on code in libnvdimm development tree
Simplified queue_work_near to just convert node into a CPU Split up drivers core and PM core patchesv1->v2:
Renamed queue_work_near to queue_work_node
Added WARN_ON_ONCE if we use queue_work_node with per-cpu workqueuev2->v3:
Added Acked-by for queue_work_node patch
Continued rename from _near to _node to be consistent with queue_work_node
Renamed async_schedule_near_domain to async_schedule_node_domain Renamed async_schedule_near to async_schedule_node
Added kerneldoc for new async_schedule_XXX functions
Updated patch description for patch 4 to include data on potential gainsv3->v4
Added patch to consolidate use of need_parent_lock
Make asynchronous driver probing explicit about use of drvdata Added Acked-by for PM core patch
---
Alexander Duyck (6):
workqueue: Provide queue_work_node to queue work near a given NUMA node
async: Add support for queueing on specific NUMA node device core: Consolidate locking and unlocking of parent and device driver core: Probe devices asynchronously instead of the driver driver core: Attach devices on CPU local to device node
PM core: Use new async_schedule_dev command
drivers/base/base.h | 2 + drivers/base/bus.c | 46 ++-------------- drivers/base/dd.c | 130 ++++++++++++++++++++++++++++++++++++++++----- drivers/base/power/main.c | 12 ++-- include/linux/async.h | 84 ++++++++++++++++++++++++++++- include/linux/device.h | 4 + include/linux/workqueue.h | 2 + kernel/async.c | 53 ++++++++++-------- kernel/workqueue.c | 84 +++++++++++++++++++++++++++++ 9 files changed, 329 insertions(+), 88 deletions(-)
--
PATCH v4 1/6 workqueue: Provide queue_work_node to queue work near a given NUMA node
This patch provides a new function queue_work_node which is meant to schedule work on a "random" CPU of the requested NUMA node. The main motivation for this is to help assist asynchronous init to better improve boot times for devices that are local to a specific node.
For now we just default to the first CPU that is in the intersection of the cpumask of the node and the online cpumask. The only exception is if the CPU is local to the node we will just use the current CPU. This should work for our purposes as we are currently only using this for unbound work so the CPU will be translated to a node anyway instead of being directly used.
As we are only using the first CPU to represent the NUMA node for now I am limiting the scope of the function so that it can only be used with unbound workqueues.
--- include/linux/workqueue.h | 2 + kernel/workqueue.c | 84 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 86 insertions(+)
include/linux/workqueue.h +2 -0
@@ -463,6 +463,8 @@ int apply_workqueue_attrs(struct workqueue_struct *wq,extern bool queue_work_on (int cpu,struct workqueue_struct *wq,struct work_struct *work); +extern bool queue_work_node (int node,struct workqueue_struct *wq, +struct work_struct *work);extern bool queue_delayed_work_on (int cpu,struct workqueue_struct *wq,struct delayed_work *work,unsigned long delay); extern bool mod_delayed_work_on(int cpu, struct workqueue_struct *wq,
kernel/workqueue.c +84 -0
90 lines not on this page · raw message
PATCH v4 2/6 async: Add support for queueing on specific NUMA node
In reply to Alexander DuyckThis patch introduces four new variants of the async_schedule_ functions that allow scheduling on a specific NUMA node.
The first two functions are async_schedule_near and async_schedule_near_domain which end up mapping to async_schedule and async_schedule_domain but provide NUMA node specific functionality. They replace the original functions which were moved to inline function definitions that call the new functions while passing NUMA_NO_NODE.
The second two functions are async_schedule_dev and async_schedule_dev_domain which provide NUMA specific functionality when passing a device as the data member and that device has a NUMA node other than NUMA_NO_NODE.
The main motivation behind this is to address the need to be able to schedule device specific init work on specific NUMA nodes in order to improve performance of memory initialization.
--- include/linux/async.h | 84 +++++++++++++++++++++++++++++++++++++++++++++++-- kernel/async.c | 53 +++++++++++++++++-------------- 2 files changed, 110 insertions(+), 27 deletions(-)
include/linux/async.h +81 -3
@@ -14,6 +14,8 @@#include <linux/types.h> #include <linux/list.h> +#include <linux/numa.h> +#include <linux/device.h> typedef u64 async_cookie_t ;typedef void (*async_func_t ) (void *data,async_cookie_t cookie);
@@ -37,9 +39,85 @@ struct async_domain {struct async_domain _name = { .pending =LIST_HEAD_INIT (_name.pending), \ .registered = 0 } -extern async_cookie_t async_schedule (async_func_t func,void *data); -extern async_cookie_t async_schedule_domain (async_func_t func,void *data, -struct async_domain *domain); +async_cookie_t async_schedule_node (async_func_t func,void *data, +int node); +async_cookie_t async_schedule_node_domain (async_func_t func,void *data, +int node, +struct async_domain *domain); + +/** +* async_schedule - schedule a function for asynchronous execution +* @func: function to execute asynchronously +* @data: data pointer to pass to the function +* +* Returns an async_cookie_t that may be used for checkpointing later. +* Note: This function may be called from atomic or non-atomic contexts. +*/ +static inline async_cookie_t async_schedule (async_func_t func,void *data) +{ +return async_schedule_node (func, data,NUMA_NO_NODE ); +} + +/** +* async_schedule_domain - schedule a function for asynchronous execution within a certain domain +* @func: function to execute asynchronously +* @data: data pointer to pass to the function +* @domain: the domain +* +* Returns an async_cookie_t that may be used for checkpointing later. +* @domain may be used in the async_synchronize_*_domain() functions to +* wait within a certain synchronization domain rather than globally. A +* synchronization domain is specified via @domain. +* Note: This function may be called from atomic or non-atomic contexts. +*/ +static inline async_cookie_t +async_schedule_domain (async_func_t func,void *data, +struct async_domain *domain) +{ +return async_schedule_node_domain (func, data,NUMA_NO_NODE , domain); +} + +/** +* async_schedule_dev - A device specific version of async_schedule +* @func: function to execute asynchronously +* @dev: device argument to be passed to function +* +* Returns an async_cookie_t that may be used for checkpointing later. +* @dev is used as both the argument for the function and to provide NUMA +* context for where to run the function. By doing this we can try to +* provide for the best possible outcome by operating on the device on the +* CPUs closest to the device. +* Note: This function may be called from atomic or non-atomic contexts. +*/ +static inline async_cookie_t +async_schedule_dev (async_func_t func,struct device *dev) +{ +return async_schedule_node (func, dev,dev_to_node (dev)); +} + +/** +* async_schedule_dev_domain - A device specific version of async_schedule_domain +* @func: function to execute asynchronously +* @dev: device argument to be passed to function +* @domain: the domain +* +* Returns an async_cookie_t that may be used for checkpointing later. +* @dev is used as both the argument for the function and to provide NUMA +* context for where to run the function. By doing this we can try to +* provide for the best possible outcome by operating on the device on the +* CPUs closest to the device. +* @domain may be used in the async_synchronize_*_domain() functions to +* wait within a certain synchronization domain rather than globally. A +* synchronization domain is specified via @domain. +* Note: This function may be called from atomic or non-atomic contexts. +*/ +static inline async_cookie_t +async_schedule_dev_domain (async_func_t func,struct device *dev, +struct async_domain *domain) +{ +return async_schedule_node_domain (func, dev,dev_to_node (dev), domain); +} +void async_unregister_domain (struct async_domain *domain);extern void async_synchronize_full (void );extern void async_synchronize_full_domain (struct async_domain *domain);
kernel/async.c +29 -24
79 lines not on this page · raw message
PATCH v4 3/6 device core: Consolidate locking and unlocking of parent and device
In reply to Alexander DuyckThis patch is meant to try and consolidate all of the locking and unlocking of both the parent and device when attaching or removing a driver from a given device.
To do that I first consolidated the lock pattern into two functions __device_driver_lock and __device_driver_unlock. After doing that I then created functions specific to attaching and detaching the driver while acquiring this locks. By doing this I was able to reduce the number of spots where we touch need_parent_lock from 12 down to 4.
--- drivers/base/base.h | 2 + drivers/base/bus.c | 23 ++------------ drivers/base/dd.c | 83 ++++++++++++++++++++++++++++++++++++++++++--------- 3 files changed, 75 insertions(+), 33 deletions(-)
drivers/base/base.h +2 -0
@@ -124,6 +124,8 @@ extern int driver_add_groups(struct device_driver *drv, const struct attribute_group **groups);extern void driver_remove_groups (struct device_driver *drv,const struct attribute_group **groups); +int device_driver_attach (struct device_driver *drv,struct device *dev); +void device_driver_detach (struct device *dev);extern char *make_class_name (const char *name,struct kobject *kobj);
drivers/base/bus.c +4 -19
@@ -184,11 +184,7 @@ static ssize_t unbind_store(struct device_driver *drv, const char *buf, dev =bus_find_device_by_name (bus,NULL , buf);if (dev && dev->driver == drv) { -if (dev->parent && dev->bus->need_parent_lock) -device_lock (dev->parent); -device_release_driver (dev); -if (dev->parent && dev->bus->need_parent_lock) -device_unlock (dev->parent); +device_driver_detach (dev); err = count; }put_device (dev);
@@ -211,13 +207,7 @@ static ssize_t bind_store(struct device_driver *drv, const char *buf, dev =bus_find_device_by_name (bus,NULL , buf);if (dev && dev->driver ==NULL &&driver_match_device (drv, dev)) { -if (dev->parent && bus->need_parent_lock) -device_lock (dev->parent); -device_lock (dev); - err =driver_probe_device (drv, dev); -device_unlock (dev); -if (dev->parent && bus->need_parent_lock) -device_unlock (dev->parent); + err =device_driver_attach (drv, dev);if (err >0 ) {/* success */
@@ -769,13 +759,8 @@ int bus_rescan_devices(struct bus_type *bus)*/ int device_reprobe (struct device *dev) { -if (dev->driver) { -if (dev->parent && dev->bus->need_parent_lock) -device_lock (dev->parent); -device_release_driver (dev); -if (dev->parent && dev->bus->need_parent_lock) -device_unlock (dev->parent); - } +if (dev->driver) +device_driver_detach (dev);return bus_rescan_devices_helper (dev,NULL ); }EXPORT_SYMBOL_GPL (device_reprobe);
drivers/base/dd.c +69 -14
111 lines not on this page · raw message
Re: [driver-core PATCH v4 3/6] device core: Consolidate locking and unlocking of parent and device
Quoted 5 lines · Alexander Duyck
This patch is meant to try and consolidate all of the locking and unlocking of both the parent and device when attaching or removing a driver from a given device.
Quoted 7 lines
To do that I first consolidated the lock pattern into two functions
__device_driver_lockand__device_driver_unlock. After doing that I then created functions specific to attaching and detaching the driver while acquiring this locks. By doing this I was able to reduce the number of spots where we touchneed_parent_lockfrom 12 down to 4.Signed-off-by Alexander Duyck <[email protected]>
Quoted 187 lines
--- drivers/base/base.h | 2 + drivers/base/bus.c | 23 ++------------ drivers/base/dd.c | 83 ++++++++++++++++++++++++++++++++++++++++++--------- 3 files changed, 75 insertions(+), 33 deletions(-)drivers/base/base.h +2
-0@@ -124,6 +124,8 @@ extern int driver_add_groups(struct device_driver *drv, const struct attribute_group **groups);extern void driver_remove_groups (struct device_driver *drv,const struct attribute_group **groups); +int device_driver_attach (struct device_driver *drv,struct device *dev); +void device_driver_detach (struct device *dev);extern char *make_class_name (const char *name,struct kobject *kobj);drivers/base/bus.c +4
-19@@ -184,11 +184,7 @@ static ssize_t unbind_store(struct device_driver *drv, const char *buf, dev =bus_find_device_by_name (bus,NULL , buf);if (dev && dev->driver == drv) { -if (dev->parent && dev->bus->need_parent_lock) -device_lock (dev->parent); -device_release_driver (dev); -if (dev->parent && dev->bus->need_parent_lock) -device_unlock (dev->parent); +device_driver_detach (dev); err = count; }put_device (dev);@@ -211,13 +207,7 @@ static ssize_t bind_store(struct device_driver *drv, const char *buf, dev =bus_find_device_by_name (bus,NULL , buf);if (dev && dev->driver ==NULL &&driver_match_device (drv, dev)) { -if (dev->parent && bus->need_parent_lock) -device_lock (dev->parent); -device_lock (dev); - err =driver_probe_device (drv, dev); -device_unlock (dev); -if (dev->parent && bus->need_parent_lock) -device_unlock (dev->parent); + err =device_driver_attach (drv, dev);if (err >0 ) {/* success */ @@ -769,13 +759,8 @@ int bus_rescan_devices(struct bus_type *bus)*/ int device_reprobe (struct device *dev) { -if (dev->driver) { -if (dev->parent && dev->bus->need_parent_lock) -device_lock (dev->parent); -device_release_driver (dev); -if (dev->parent && dev->bus->need_parent_lock) -device_unlock (dev->parent); - } +if (dev->driver) +device_driver_detach (dev);return bus_rescan_devices_helper (dev,NULL ); }EXPORT_SYMBOL_GPL (device_reprobe);drivers/base/dd.c +69
-14@@ -864,6 +864,60 @@ void device_initial_probe(struct device *dev)__device_attach (dev, true); } +/* +* __device_driver_lock - acquire locks needed to manipulate dev->drv +* @dev: Device we will update driver info for +* @parent: Parent device needed if the bus requires parent lock +* +* This function will take the required locks for manipulating dev->drv. +* Normally this will just be the @dev lock, but when called for a USB +* interface, @parent lock will be held as well. +*/ +static void __device_driver_lock (struct device *dev,struct device *parent) +{ +if (parent && dev->bus->need_parent_lock) +device_lock (parent); +device_lock (dev); +} + +/* +* __device_driver_lock - release locks needed to manipulate dev->drv +* @dev: Device we will update driver info for +* @parent: Parent device needed if the bus requires parent lock +* +* This function will release the required locks for manipulating dev->drv. +* Normally this will just be the the @dev lock, but when called for a +* USB interface, @parent lock will be released as well. +*/ +static void __device_driver_unlock (struct device *dev,struct device *parent) +{ +device_unlock (dev); +if (parent && dev->bus->need_parent_lock) +device_unlock (parent); +} + +/** +* device_driver_attach - attach a specific driver to a specific device +* @drv: Driver to attach +* @dev: Device to attach it to +* +* Manually attach driver to a device. Will acquire both @dev lock and +* @dev->parent lock if needed. +*/ +int device_driver_attach (struct device_driver *drv,struct device *dev) +{ +int ret =0 ; + +__device_driver_lock (dev, dev->parent); + +if (!dev->driver) + ret =driver_probe_device (drv, dev); + +__device_driver_unlock (dev, dev->parent); + +return ret; +} +static int __driver_attach (struct device *dev,void *data) {struct device_driver *drv = data;@@ -891,14 +945,7 @@ static int __driver_attach(struct device *dev, void *data)return ret; }/* ret > 0 means positive match */ -if (dev->parent && dev->bus->need_parent_lock) -device_lock (dev->parent); -device_lock (dev); -if (!dev->driver) -driver_probe_device (drv, dev); -device_unlock (dev); -if (dev->parent && dev->bus->need_parent_lock) -device_unlock (dev->parent); +device_driver_attach (drv, dev);return 0 ; }@@ -993,16 +1040,12 @@ void device_release_driver_internal(struct device *dev,struct device_driver *drv, struct device *parent) { -if (parent && dev->bus->need_parent_lock) -device_lock (parent); +__device_driver_lock (dev ,parent ); -device_lock (dev);if (!drv || drv == dev->driver)__device_release_driver (dev, parent); -device_unlock (dev); -if (parent && dev->bus->need_parent_lock) -device_unlock (parent); +__device_driver_unlock (dev, parent); }/** @@ -1028,6 +1071,18 @@ void device_release_driver(struct device *dev)EXPORT_SYMBOL_GPL (device_release_driver);/** +* device_driver_detach - detach driver from a specific device +* @dev: device to detach driver from +* +* Manually detach driver from device. Will acquire both @dev lock and +* @dev->parent lock if needed. +*/ +void device_driver_detach (struct device *dev) +{ +device_release_driver_internal (dev,NULL , dev->parent); +} + +/** * driver_detach - detach driver from all devices it controls. * @drv: driver. */
Re: [driver-core PATCH v4 3/6] device core: Consolidate locking and unlocking of parent and device
In reply to Alexander DuyckQuoted 8 lines · Alexander Duyck
This patch is meant to try and consolidate all of the locking and unlocking of both the parent and device when attaching or removing a driver from a given device.
To do that I first consolidated the lock pattern into two functions
__device_driver_lockand__device_driver_unlock. After doing that I then created functions specific to attaching and detaching the driver while acquiring this locks. By doing this I was able to reduce the number of
^^^^Should "this"perhaps be changed into "these"
Quoted 3 lines
+-/*>+- *__device_driver_lock - acquire locks needed to manipulate dev->drv +- *@dev: Device we will update driver info for +- *@parent: Parent device needed if the bus requires parent lock
Please consider splitting that description into two sentences to enhance clarity, e.g. "Parent device. Needed if the bus requires parent lock."
Quoted 1 line
+- *@parent: Parent device needed if the bus requires parent lock
Same comment here.
Quoted 10 lines
/**>+-
*device_driver_detach- detach driver from a specific device+- *@dev: device to detach driver from +- *>+- *Manually detach driver from device. Will acquire both @dev lock and +- *@dev->parent lock if needed. +- */ +-void device_driver_detach(struct device *dev) +-{>+- device_release_driver_internal(dev, NULL, dev->parent);>+-}>+- +-/**> *driver_detach - detach driver from all devices it controls.*@drv: driver.
*/
Elsewhere in the driver core the word "manually"only appears in comments above exported functions and functions that implement sysfs store methods. Since device_driver_detach() is neither, I think the word "manually"should not be used in the comment block above that function. But since the rest of this patch looks fine to me, feel free to add:
PATCH v4 4/6 driver core: Probe devices asynchronously instead of the driver
In reply to Alexander DuyckThis change makes it so that we probe devices asynchronously instead of the driver. This results in us seeing the same behavior if the device is registered before the driver or after. This way we can avoid serializing the initialization should the driver not be loaded until after the devices have already been added.
The motivation behind this is that if we have a set of devices that take a significant amount of time to load we can greatly reduce the time to load by processing them in parallel instead of one at a time. In addition, each device can exist on a different node so placing a single thread on one CPU to initialize all of the devices for a given driver can result in poor performance on a system with multiple nodes.
I am using the driver_data member of the device struct to store the driver pointer while we wait on the deferred probe call. This should be safe to do as the value will either be set to NULL on a failed probe or driver load followed by unload, or the driver value itself will be set on a successful driver load.
--- drivers/base/bus.c | 23 +++-------------------- drivers/base/dd.c | 45 +++++++++++++++++++++++++++++++++++++++++++++ include/linux/device.h | 4 +++- 3 files changed, 51 insertions(+), 21 deletions(-)
drivers/base/bus.c +3 -20
@@ -606,17 +606,6 @@ static ssize_t uevent_store(struct device_driver *drv, const char *buf, }static DRIVER_ATTR_WO (uevent); -static void driver_attach_async (void *_drv,async_cookie_t cookie) -{ -struct device_driver *drv = _drv; -int ret; - - ret =driver_attach (drv); - -pr_debug ("bus: '%s': driver %s async attach completed: %d\n" , - drv->bus->name, drv->name, ret); -} -/** * bus_add_driver - Add a driver to the bus. * @drv: driver.
@@ -649,15 +638,9 @@ int bus_add_driver(struct device_driver *drv)klist_add_tail (&priv->knode_bus, &bus->p->klist_drivers);if (drv->bus->p->drivers_autoprobe) { -if (driver_allows_async_probing (drv)) { -pr_debug ("bus: '%s': probing driver %s asynchronously\n" , - drv->bus->name, drv->name); -async_schedule (driver_attach_async, drv); - }else { - error =driver_attach (drv); -if (error) - goto out_unregister; - } + error =driver_attach (drv); +if (error) + goto out_unregister; }module_add_driver (drv->owner, drv);
drivers/base/dd.c +45 -0
63 lines not on this page · raw message
include/linux/device.h +3 -1
@@ -906,7 +906,9 @@ struct dev_links_info {* variants, which GPIO pins act in what additional roles, and so * on. This shrinks the "Board Support Packages" (BSPs) and * minimizes board-specific #ifdefs in drivers. -* @driver_data: Private pointer for driver specific info. +* @driver_data: Private pointer for driver specific info if driver is +* non-NULL. Pointer to deferred driver to be attached if driver +* is NULL. * @links: Links to suppliers and consumers of this device. * @power: For device power management. * See Documentation/driver-api/pm/devices.rst for details.
Re: [driver-core PATCH v4 4/6] driver core: Probe devices asynchronously instead of the driver
Quoted 17 lines · Alexander Duyck
+-static void __driver_attach_async_helper(void *_dev, async_cookie_t cookie) +-{>+- struct device *dev =_dev;>+- +- __device_driver_lock(dev, dev->parent);>+- +- /*>+- *If someone attempted to bind a driver either successfully or +- *unsuccessfully before we got here we should just skip the driver +- *probe call. +- */ +- if (!dev->driver) {>+- struct device_driver *drv =dev_get_drvdata(dev);>+- +- if (drv) +- driver_probe_device(drv, dev);>+- }>+- +- __device_driver_unlock(dev, dev->parent);>+- +- put_device(dev);>+- +- dev_dbg(dev, "async probe completed\n";>+-}>+-static int
__driver_attach(struct device *dev, void *data)
{> structdevice_driver*drv =data;>@@-945,6 +-971,25 @@static int__driver_attach(struct device *dev, void *data)
return ret;> }/*ret >0 means positive match */
Quoted 7 lines
+- if (driver_allows_async_probing(drv)) {>+- /*>+- *Instead of probing the device synchronously we will
+- *probe it asynchronously to allow for more parallelism.
+- *>+- *We only take the device lock here in order to guarantee
+- *that the dev->driver and driver_data fields are protected
+- */
+- dev_dbg(dev, "scheduling asynchronous probe\n";>+- device_lock(dev);>+- if (!dev->driver) {>+- get_device(dev);>+- dev_set_drvdata(dev, drv);>+- async_schedule(__driver_attach_async_helper, dev);>+- }>+- device_unlock(dev);>+- return 0;>+- }>+-
device_driver_attach(drv, dev);What prevents that the driver pointer becomes invalid after async_schedule() has been called and before __driver_attach_async_helper() is called? I think we need protection against concurrent driver_unregister() and __driver_attach_async_helper() calls. I'm not sure whether that is possible without introducing a new mutex.
Thanks,
Bart.
Re: [driver-core PATCH v4 4/6] driver core: Probe devices asynchronously instead of the driver
Quoted 12 lines · Bart Van Assche
drivers/base/dd.c int device_driver_attach(struct device_driver *drv, struct device *dev) · line 932
Quoted 11 lines · Alexander Duyck
drivers/base/dd.c +11
-0+static void __driver_attach_async_helper (void *_dev,async_cookie_t cookie) +{ +struct device *dev = _dev; + +__device_driver_lock (dev, dev->parent); + +/* +* If someone attempted to bind a driver either successfully or +* unsuccessfully before we got here we should just skip the driver +* probe call. +*/
The answer to your question below is up here.
Quoted 47 lines
Quoted 31 lines
drivers/base/dd.c +24
-0+if (!dev->driver) { +struct device_driver *drv =dev_get_drvdata (dev); + +if (drv) +driver_probe_device (drv, dev); + } + +__device_driver_unlock (dev, dev->parent); + +put_device (dev); + +dev_dbg (dev,"async probe completed\n" ); +} +static int __driver_attach (struct device *dev,void *data) {struct device_driver *drv = data;@@ -945,6 +971,25 @@ static int __driver_attach(struct device *dev, void *data)return ret; }/* ret > 0 means positive match */ +if (driver_allows_async_probing (drv)) { +/* +* Instead of probing the device synchronously we will +* probe it asynchronously to allow for more parallelism. +* +* We only take the device lock here in order to guarantee +* that the dev->driver and driver_data fields are protected +*/ +dev_dbg (dev,"scheduling asynchronous probe\n" ); +device_lock (dev);drivers/base/dd.c static int __driver_attach(struct device *dev, void *data) · line 993
Quoted 10 lines
drivers/base/dd.c +9
-0+if (!dev->driver) { +get_device (dev); +dev_set_drvdata (dev, drv); +async_schedule (__driver_attach_async_helper, dev); + } +device_unlock (dev); +return 0 ; + } +device_driver_attach (drv, dev);What prevents that the driver pointer becomes invalid after
async_schedule()has been called and before__driver_attach_async_helper()is called? I think we need protection against concurrentdriver_unregister()and__driver_attach_async_helper()calls. I'm not sure whether that is possible without introducing a new mutex.
Quoted 3 lines
Thanks,
Bart.
See the spot called out above.
Basically if somebody loads a driver the dev->driver becomes set. If a driver is removed it will clear dev->driver and set driver_data to 0/NULL. That is what I am using as a mutex to track it in conjunction with the device mutex. Basically if somebody attempts to attach a driver before we get there we just exit and don't attempt to load this driver.
Re: [driver-core PATCH v4 4/6] driver core: Probe devices asynchronously instead of the driver
Quoted 5 lines · Alexander Duyck
Basically if somebody loads a driver the
dev->driverbecomes set. If a driver is removed it will cleardev->driverand setdriver_datato 0/NULL. That is what I am using as a mutex to track it in conjunction with the device mutex. Basically if somebody attempts to attach a driver before we get there we just exit and don't attempt to load this driver.
I don't think that the above matches your code. __device_attach() does not set the dev->driver pointer before scheduling an asynchronous probe. Only dev->driver_data gets set before the asynchonous probe is scheduled. Since driver_detach() only iterates over devices that are in the per-driver klist it will skip all devices for which an asynchronous probe has been scheduled but __device_attach_async_helper() has not yet been called. My conclusion remains that this patch does not prevent a driver pointer to become invalid concurrently with __device_attach_async_helper() dereferencing the same driver pointer.
Bart.
Re: [driver-core PATCH v4 4/6] driver core: Probe devices asynchronously instead of the driver
Quoted 18 lines · Bart Van Assche
Quoted 5 lines · Alexander Duyck
Basically if somebody loads a driver the
dev->driverbecomes set. If a driver is removed it will cleardev->driverand setdriver_datato 0/NULL. That is what I am using as a mutex to track it in conjunction with the device mutex. Basically if somebody attempts to attach a driver before we get there we just exit and don't attempt to load this driver.I don't think that the above matches your code.
__device_attach()does not set thedev->driverpointer before scheduling an asynchronous probe. Onlydev->driver_datagets set before the asynchonous probe is scheduled. Sincedriver_detach()only iterates over devices that are in the per-driver klist it will skip all devices for which an asynchronous probe has been scheduled but__device_attach_async_helper()has not yet been called. My conclusion remains that this patch does not prevent a driver pointer to become invalid concurrently with__device_attach_async_helper()dereferencing the same driver pointer.
Quoted 1 line
Bart.
I see what you are talking about now. Actually I think this was an existing issue before my patch even came into play. Basically the code as it currently stands is device specific in terms of the attach and release code.
I wonder if we shouldn't have the async_synchronize_full call in __device_release_driver moved down and into driver_detach before we even start the for loop. Assuming the driver is no longer associated with the bus that should flush out all devices so that we can then pull them out of the devices list at least. I may look at adding an additional bitflag to the device struct to indicate that it has a driver attach pending. Then for things like races between any attach and detach calls the logic becomes pretty straight forward. Attach will set the bit and provide driver data, detach will clear the bit and the driver data. If a driver loads in between it should clear the bit as well.
I'll work on it over the next couple days and hopefully have something ready for testing/review early next week.
Thanks.
- Alex
Re: [driver-core PATCH v4 4/6] driver core: Probe devices asynchronously instead of the driver
Quoted 17 lines · Alexander Duyck
I see what you are talking about now. Actually I think this was an existing issue before my patch even came into play. Basically the code as it currently stands is device specific in terms of the attach and release code.
I wonder if we shouldn't have the
async_synchronize_fullcall in__device_release_drivermoved down and intodriver_detachbefore we even start the for loop. Assuming the driver is no longer associated with the bus that should flush out all devices so that we can then pull them out of the devices list at least. I may look at adding an additional bitflag to the device struct to indicate that it has a driver attach pending. Then for things like races between any attach and detach calls the logic becomes pretty straight forward. Attach will set the bit and provide driver data, detach will clear the bit and the driver data. If a driver loads in between it should clear the bit as well.
Quoted 2 lines
I'll work on it over the next couple days and hopefully have something ready for testing/review early next week.
Hi Alex,
How about checking in __driver_attach_async_helper() whether the driver pointer is still valid by checking whether bus_for_each_drv(dev->bus, ...) can still find the driver pointer? That approach requires protection with a mutex to avoid races with the driver detach code but shouldn't require any new flags in struct device.
Thanks,
Bart.
Re: [driver-core PATCH v4 4/6] driver core: Probe devices asynchronously instead of the driver
Quoted 33 lines · Bart Van Assche
Quoted 22 lines · Alexander Duyck
I see what you are talking about now. Actually I think this was an existing issue before my patch even came into play. Basically the code
as it currently stands is device specific in terms of the attach and
release code.I wonder if we shouldn't have the
async_synchronize_fullcall in__device_release_drivermoved down and intodriver_detachbefore we even start the for loop. Assuming the driver is no longer associated
with the bus that should flush out all devices so that we can then pull them out of the devices list at least. I may look at adding an additional bitflag to the device struct to indicate that it has a driver attach pending. Then for things like races between any attach
and detach calls the logic becomes pretty straight forward. Attach will set the bit and provide driver data, detach will clear the bit and the driver data. If a driver loads in between it should clear the
bit as well.Quoted 3 lines
I'll work on it over the next couple days and hopefully have something
ready for testing/review early next week.Hi Alex,
How about checking in
__driver_attach_async_helper()whether the driver
pointer is still valid by checking whether bus_for_each_drv(dev-Quoted 1 line
bus,
Quoted 8 lines
...) can still find the driver pointer? That approach requires protection with a mutex to avoid races with the driver detach code but
shouldn't require any new flags in struct device.Thanks,
Bart.
That doesn't solve the problem I was pointing out though.
So the issue you are addressing by rechecking the bus should already be handled by just calling async_synchronize_full in driver_detach. After all we can't have a driver that is being added to the bus while it is also being removed. So if we are detaching the driver calling async_synchronize_full will flush out any deferred attach calls and there will be no further calls since the driver has already been removed from the bus.
The issue I was thinking of is how do we deal with races between device_attach and device_release_driver. In that case we know the device we want to remove a driver from, but we may not have information about the driver. The easiest solution is to basically just disable the pending enable. I could use the approach I am doing now and just NULL out the driver_data if dev->driver is NULL. The only thing I am thinking about is if just dev->driver being NULL is enough to signal that we are using driver_data to carry a pointer to a pending driver, or if we should add an extra bit to carry that meaning. It would be pretty easy to just add a bit and then use that to prevent any false reads of the deferred driver as driver data, or driver data as a deferred driver as it would essentially act as a type bit.
Thanks.
- Alex
PATCH v4 5/6 driver core: Attach devices on CPU local to device node
In reply to Alexander DuyckThis change makes it so that we call the asynchronous probe routines on a CPU local to the device node. By doing this we should be able to improve our initialization time significantly as we can avoid having to access the device from a remote node which may introduce higher latency.
For example, in the case of initializing memory for NVDIMM this can have a singifcant impact as initialing 3TB on remote node can take up to 39 seconds while initialing it on a local node only takes 23 seconds. It is situations like this where we will see the biggest improvement.
--- drivers/base/dd.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/base/dd.c +2 -2
@@ -827,7 +827,7 @@ static int __device_attach(struct device *dev, bool allow_async)*/ dev_dbg (dev,"scheduling asynchronous probe\n" );get_device (dev); -async_schedule (__device_attach_async_helper, dev); +async_schedule_dev (__device_attach_async_helper, dev); }else {pm_request_idle (dev); }
@@ -984,7 +984,7 @@ static int __driver_attach(struct device *dev, void *data)if (!dev->driver) {get_device (dev);dev_set_drvdata (dev, drv); -async_schedule (__driver_attach_async_helper, dev); +async_schedule_dev (__driver_attach_async_helper, dev); }device_unlock (dev);return 0 ;
PATCH v4 6/6 PM core: Use new async_schedule_dev command
In reply to Alexander DuyckThis change makes it so that we use the device specific version of the async_schedule commands to defer various tasks related to power management. By doing this we should see a slight improvement in performance as any device that is sensitive to latency/locality in the setup will now be initializing on the node closest to the device.
--- drivers/base/power/main.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
drivers/base/power/main.c +6 -6
@@ -726,7 +726,7 @@ void dpm_noirq_resume_devices(pm_message_t state)reinit_completion (&dev->power.completion);if (is_async (dev)) {get_device (dev); -async_schedule (async_resume_noirq, dev); +async_schedule_dev (async_resume_noirq, dev); } }
@@ -883,7 +883,7 @@ void dpm_resume_early(pm_message_t state)reinit_completion (&dev->power.completion);if (is_async (dev)) {get_device (dev); -async_schedule (async_resume_early, dev); +async_schedule_dev (async_resume_early, dev); } }
@@ -1047,7 +1047,7 @@ void dpm_resume(pm_message_t state)reinit_completion (&dev->power.completion);if (is_async (dev)) {get_device (dev); -async_schedule (async_resume, dev); +async_schedule_dev (async_resume, dev); } }
@@ -1366,7 +1366,7 @@ static int device_suspend_noirq(struct device *dev)if (is_async (dev)) {get_device (dev); -async_schedule (async_suspend_noirq, dev); +async_schedule_dev (async_suspend_noirq, dev);return 0 ; }return __device_suspend_noirq (dev, pm_transition, false);
@@ -1569,7 +1569,7 @@ static int device_suspend_late(struct device *dev)if (is_async (dev)) {get_device (dev); -async_schedule (async_suspend_late, dev); +async_schedule_dev (async_suspend_late, dev);return 0 ; }
@@ -1833,7 +1833,7 @@ static int device_suspend(struct device *dev)if (is_async (dev)) {get_device (dev); -async_schedule (async_suspend, dev); +async_schedule_dev (async_suspend, dev);return 0 ; }