Add NUMA aware async_schedule calls
Subsystems (5)
v2 cover Add NUMA aware async_schedule calls
This patch set provides functionality that will help to improve the locality of the async_schedule calls used to provide deferred initialization.
This patch set originally started out with me focused on just the one call to async_schedule_domain in the nvdimm tree that was being used to defer the device_add call however after doing some digging I realized the scope of this was much broader than I had originally planned. As such I went through and reworked the underlying infrastructure down to replacing the queue_work call itself with a function of my own and opted to try and provide a NUMA aware solution that would work for a broader audience.
RFC->v1:
Dropped nvdimm patch to submit later.
It relies on code in libnvdimm development tree.
Simplified queue_work_near to just convert node into a CPU.
Split up drivers core and PM core patches.v1->v2:
Renamed queue_work_near to queue_work_node
Added WARN_ON_ONCE if we use queue_work_node with per-cpu workqueue
---
Alexander Duyck (5):
workqueue: Provide queue_work_node to queue work near a given NUMA node
async: Add support for queueing on specific NUMA node driver core: Probe devices asynchronously instead of the driver driver core: Attach devices on CPU local to device node
PM core: Use new async_schedule_dev command
drivers/base/bus.c | 23 ++---------- drivers/base/dd.c | 44 +++++++++++++++++++++++- drivers/base/power/main.c | 12 +++--- include/linux/async.h | 36 ++++++++++++++++++- include/linux/workqueue.h | 2 + kernel/async.c | 47 ++++++++++++------------- kernel/workqueue.c | 84 +++++++++++++++++++++++++++++++++++++++++++++ 7 files changed, 193 insertions(+), 55 deletions(-)
--
PATCH v2 1/5 workqueue: Provide queue_work_node to queue work near a given NUMA node
This patch provides a new function queue_work_node which is meant to schedule work on a "random" CPU of the requested NUMA node. The main motivation for this is to help assist asynchronous init to better improve boot times for devices that are local to a specific node.
For now we just default to the first CPU that is in the intersection of the cpumask of the node and the online cpumask. The only exception is if the CPU is local to the node we will just use the current CPU. This should work for our purposes as we are currently only using this for unbound work so the CPU will be translated to a node anyway instead of being directly used.
As we are only using the first CPU to represent the NUMA node for now I am limiting the scope of the function so that it can only be used with unbound workqueues.
---
RFC->v1:
Simplified queue_work_near to just convert node into a CPU.v1->v2:
Renamed queue_work_near to queue_work_node
Added WARN_ON_ONCE if we use queue_work_node with per-cpu workqueue
include/linux/workqueue.h | 2 + kernel/workqueue.c | 84 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 86 insertions(+)
include/linux/workqueue.h +2 -0
@@ -463,6 +463,8 @@ int apply_workqueue_attrs(struct workqueue_struct *wq,extern bool queue_work_on (int cpu,struct workqueue_struct *wq,struct work_struct *work); +extern bool queue_work_node (int node,struct workqueue_struct *wq, +struct work_struct *work);extern bool queue_delayed_work_on (int cpu,struct workqueue_struct *wq,struct delayed_work *work,unsigned long delay); extern bool mod_delayed_work_on(int cpu, struct workqueue_struct *wq,
kernel/workqueue.c +84 -0
90 lines not on this page · raw message
Re: [workqueue/driver-core PATCH v2 1/5] workqueue: Provide queue_work_node to queue work near a given NUMA node
Quoted 11 lines · Alexander Duyck
This patch provides a new function
queue_work_nodewhich is meant to schedule work on a "random" CPU of the requested NUMA node. The main motivation for this is to help assist asynchronous init to better improve boot times for devices that are local to a specific node.For now we just default to the first CPU that is in the intersection of the cpumask of the node and the online cpumask. The only exception is if the CPU is local to the node we will just use the current CPU. This should work for our purposes as we are currently only using this for unbound work so the CPU will be translated to a node anyway instead of being directly used.
Quoted 5 lines
As we are only using the first CPU to represent the NUMA node for now I am limiting the scope of the function so that it can only be used with unbound workqueues.
Signed-off-by Alexander Duyck <[email protected]>
Please let me know how you wanna route the patch.
Thanks.
Signature · 1 line
tejun
Re: [workqueue/driver-core PATCH v2 1/5] workqueue: Provide queue_work_node to queue work near a given NUMA node
Quoted 16 lines · Tejun Heo
Quoted 15 lines · Alexander Duyck
This patch provides a new function
queue_work_nodewhich is meant to schedule work on a "random" CPU of the requested NUMA node. The main motivation for this is to help assist asynchronous init to better improve boot times for devices that are local to a specific node.For now we just default to the first CPU that is in the intersection of the cpumask of the node and the online cpumask. The only exception is if the CPU is local to the node we will just use the current CPU. This should work for our purposes as we are currently only using this for unbound work so the CPU will be translated to a node anyway instead of being directly used.
As we are only using the first CPU to represent the NUMA node for now I am limiting the scope of the function so that it can only be used with unbound workqueues.
Quoted 7 lines
Quoted 1 line
Signed-off-by Alexander Duyck <[email protected]>Acked-by Tejun Heo <[email protected]>Please let me know how you wanna route the patch.
Thanks.
I would be good with routing the patches through you if that works. I had included you, Greg, and Andrew as I wasn't sure how you guys had wanted this routed since this affected both the workqueue and device trees.
I'll update the patches to resolve the lack of kerneldoc for the new "async_" functions and add some comments to the patch descriptions on the gains seen related to some of the specific patches for v3.
Thanks.
- Alex
Greg KH
maintainerRe: [workqueue/driver-core PATCH v2 1/5] workqueue: Provide queue_work_node to queue work near a given NUMA node
Quoted 27 lines · Alexander Duyck
Quoted 16 lines · Tejun Heo
Quoted 15 lines · Alexander Duyck
This patch provides a new function
queue_work_nodewhich is meant to schedule work on a "random" CPU of the requested NUMA node. The main motivation for this is to help assist asynchronous init to better improve boot times for devices that are local to a specific node.For now we just default to the first CPU that is in the intersection of the cpumask of the node and the online cpumask. The only exception is if the CPU is local to the node we will just use the current CPU. This should work for our purposes as we are currently only using this for unbound work so the CPU will be translated to a node anyway instead of being directly used.
As we are only using the first CPU to represent the NUMA node for now I am limiting the scope of the function so that it can only be used with unbound workqueues.
Quoted 7 lines
Quoted 1 line
Signed-off-by Alexander Duyck <[email protected]>Acked-by Tejun Heo <[email protected]>Please let me know how you wanna route the patch.
Thanks.
Quoted 7 lines
I would be good with routing the patches through you if that works. I had included you, Greg, and Andrew as I wasn't sure how you guys had wanted this routed since this affected both the workqueue and device trees.
I'll update the patches to resolve the lack of kerneldoc for the new "
async_" functions and add some comments to the patch descriptions on the gains seen related to some of the specific patches for v3.
As Tejun has acked this, and it affects the driver core, I'll be glad to take it.
thanks,
greg k-h
Re: [workqueue/driver-core PATCH v2 1/5] workqueue: Provide queue_work_node to queue work near a given NUMA node
Quoted 36 lines · Greg KH
Quoted 27 lines · Alexander Duyck
Quoted 16 lines · Tejun Heo
Quoted 15 lines · Alexander Duyck
This patch provides a new function
queue_work_nodewhich is meant to schedule work on a "random" CPU of the requested NUMA node. The main motivation for this is to help assist asynchronous init to better improve boot times for devices that are local to a specific node.For now we just default to the first CPU that is in the intersection of the cpumask of the node and the online cpumask. The only exception is if the CPU is local to the node we will just use the current CPU. This should work for our purposes as we are currently only using this for unbound work so the CPU will be translated to a node anyway instead of being directly used.
As we are only using the first CPU to represent the NUMA node for now I am limiting the scope of the function so that it can only be used with unbound workqueues.
Quoted 7 lines
Quoted 1 line
Signed-off-by Alexander Duyck <[email protected]>Acked-by Tejun Heo <[email protected]>Please let me know how you wanna route the patch.
Thanks.
Quoted 7 lines
I would be good with routing the patches through you if that works. I had included you, Greg, and Andrew as I wasn't sure how you guys had wanted this routed since this affected both the workqueue and device trees.
I'll update the patches to resolve the lack of kerneldoc for the new "
async_" functions and add some comments to the patch descriptions on the gains seen related to some of the specific patches for v3.
Quoted 6 lines
As Tejun has acked this, and it affects the driver core, I'll be glad to take it.
thanks,
greg k-h
Okay. That works. I will drop Tejun and Andrew to the Cc, and include you on the To line for the next submission so it is a bit more clear on who should be applying this.
Thanks.
- Alex
PATCH v2 2/5 async: Add support for queueing on specific NUMA node
In reply to Alexander DuyckThis patch introduces four new variants of the async_schedule_ functions that allow scheduling on a specific NUMA node.
The first two functions are async_schedule_near and async_schedule_near_domain which end up mapping to async_schedule and async_schedule_domain but provide NUMA node specific functionality. They replace the original functions which were moved to inline function definitions that call the new functions while passing NUMA_NO_NODE.
The second two functions are async_schedule_dev and async_schedule_dev_domain which provide NUMA specific functionality when passing a device as the data member and that device has a NUMA node other than NUMA_NO_NODE.
The main motivation behind this is to address the need to be able to schedule device specific init work on specific NUMA nodes in order to improve performance of memory initialization.
---
v1->v2:
Replaced call to queue_work_near with call to queue_work_node
include/linux/async.h | 36 +++++++++++++++++++++++++++++++++--- kernel/async.c | 47 ++++++++++++++++++++++------------------------- 2 files changed, 55 insertions(+), 28 deletions(-)
include/linux/async.h +33 -3
@@ -14,6 +14,8 @@#include <linux/types.h> #include <linux/list.h> +#include <linux/numa.h> +#include <linux/device.h> typedef u64 async_cookie_t ;typedef void (*async_func_t ) (void *data,async_cookie_t cookie);
@@ -37,9 +39,37 @@ struct async_domain {struct async_domain _name = { .pending =LIST_HEAD_INIT (_name.pending), \ .registered = 0 } -extern async_cookie_t async_schedule (async_func_t func,void *data); -extern async_cookie_t async_schedule_domain (async_func_t func,void *data, -struct async_domain *domain); +async_cookie_t async_schedule_near (async_func_t func,void *data, +int node); +async_cookie_t async_schedule_near_domain (async_func_t func,void *data, +int node, +struct async_domain *domain); + +static inline async_cookie_t async_schedule (async_func_t func,void *data) +{ +return async_schedule_near (func, data,NUMA_NO_NODE ); +} + +static inline async_cookie_t +async_schedule_domain (async_func_t func,void *data, +struct async_domain *domain) +{ +return async_schedule_near_domain (func, data,NUMA_NO_NODE , domain); +} + +static inline async_cookie_t +async_schedule_dev (async_func_t func,struct device *dev) +{ +return async_schedule_near (func, dev,dev_to_node (dev)); +} + +static inline async_cookie_t +async_schedule_dev_domain (async_func_t func,struct device *dev, +struct async_domain *domain) +{ +return async_schedule_near_domain (func, dev,dev_to_node (dev), domain); +} +void async_unregister_domain (struct async_domain *domain);extern void async_synchronize_full (void );extern void async_synchronize_full_domain (struct async_domain *domain);
kernel/async.c +22 -25
72 lines not on this page · raw message
Greg KH
maintainerRe: [workqueue/driver-core PATCH v2 2/5] async: Add support for queueing on specific NUMA node
Quoted 68 lines · Alexander Duyck
This patch introduces four new variants of the
async_schedule_functions that allow scheduling on a specific NUMA node.The first two functions are
async_schedule_nearandasync_schedule_near_domainwhich end up mapping toasync_scheduleandasync_schedule_domainbut provide NUMA node specific functionality. They replace the original functions which were moved to inline function definitions that call the new functions while passingNUMA_NO_NODE.The second two functions are
async_schedule_devandasync_schedule_dev_domainwhich provide NUMA specific functionality when passing a device as the data member and that device has a NUMA node other thanNUMA_NO_NODE.The main motivation behind this is to address the need to be able to schedule device specific init work on specific NUMA nodes in order to improve performance of memory initialization.
Signed-off-by Alexander Duyck <[email protected]>---
v1->v2:
Replaced call toqueue_work_nearwith call toqueue_work_nodeinclude/linux/async.h | 36 +++++++++++++++++++++++++++++++++--- kernel/async.c | 47 ++++++++++++++++++++++------------------------- 2 files changed, 55 insertions(+), 28 deletions(-)include/linux/async.h +22
-3@@ -14,6 +14,8 @@#include <linux/types.h> #include <linux/list.h> +#include <linux/numa.h> +#include <linux/device.h> typedef u64 async_cookie_t ;typedef void (*async_func_t ) (void *data,async_cookie_t cookie);@@ -37,9 +39,37 @@ struct async_domain {struct async_domain _name = { .pending =LIST_HEAD_INIT (_name.pending), \ .registered = 0 } -extern async_cookie_t async_schedule (async_func_t func,void *data); -extern async_cookie_t async_schedule_domain (async_func_t func,void *data, -struct async_domain *domain); +async_cookie_t async_schedule_near (async_func_t func,void *data, +int node); +async_cookie_t async_schedule_near_domain (async_func_t func,void *data, +int node, +struct async_domain *domain); + +static inline async_cookie_t async_schedule (async_func_t func,void *data) +{ +return async_schedule_near (func, data,NUMA_NO_NODE ); +} + +static inline async_cookie_t +async_schedule_domain (async_func_t func,void *data, +struct async_domain *domain) +{ +return async_schedule_near_domain (func, data,NUMA_NO_NODE , domain); +} + +static inline async_cookie_t +async_schedule_dev (async_func_t func,struct device *dev)
include/linux/async.h struct async_domain { · line 71
Quoted 10 lines
include/linux/async.h +10
-0+{ +return async_schedule_near (func, dev,dev_to_node (dev)); +} + +static inline async_cookie_t +async_schedule_dev_domain (async_func_t func,struct device *dev, +struct async_domain *domain) +{ +return async_schedule_near_domain (func, dev,dev_to_node (dev), domain); +}
No kerneldoc for these functions? We lost it for async_schedule_domain() which used to be documented before this patch :(
thanks,
greg k-h
Re: [workqueue/driver-core PATCH v2 2/5] async: Add support for queueing on specific NUMA node
Quoted 83 lines · Greg KH
Quoted 68 lines · Alexander Duyck
This patch introduces four new variants of the
async_schedule_functions that allow scheduling on a specific NUMA node.The first two functions are
async_schedule_nearandasync_schedule_near_domainwhich end up mapping toasync_scheduleandasync_schedule_domainbut provide NUMA node specific functionality. They replace the original functions which were moved to inline function definitions that call the new functions while passingNUMA_NO_NODE.The second two functions are
async_schedule_devandasync_schedule_dev_domainwhich provide NUMA specific functionality when passing a device as the data member and that device has a NUMA node other thanNUMA_NO_NODE.The main motivation behind this is to address the need to be able to schedule device specific init work on specific NUMA nodes in order to improve performance of memory initialization.
Signed-off-by Alexander Duyck <[email protected]>---
v1->v2:
Replaced call toqueue_work_nearwith call toqueue_work_nodeinclude/linux/async.h | 36 +++++++++++++++++++++++++++++++++--- kernel/async.c | 47 ++++++++++++++++++++++------------------------- 2 files changed, 55 insertions(+), 28 deletions(-)include/linux/async.h +22
-3@@ -14,6 +14,8 @@#include <linux/types.h> #include <linux/list.h> +#include <linux/numa.h> +#include <linux/device.h> typedef u64 async_cookie_t ;typedef void (*async_func_t ) (void *data,async_cookie_t cookie);@@ -37,9 +39,37 @@ struct async_domain {struct async_domain _name = { .pending =LIST_HEAD_INIT (_name.pending), \ .registered = 0 } -extern async_cookie_t async_schedule (async_func_t func,void *data); -extern async_cookie_t async_schedule_domain (async_func_t func,void *data, -struct async_domain *domain); +async_cookie_t async_schedule_near (async_func_t func,void *data, +int node); +async_cookie_t async_schedule_near_domain (async_func_t func,void *data, +int node, +struct async_domain *domain); + +static inline async_cookie_t async_schedule (async_func_t func,void *data) +{ +return async_schedule_near (func, data,NUMA_NO_NODE ); +} + +static inline async_cookie_t +async_schedule_domain (async_func_t func,void *data, +struct async_domain *domain) +{ +return async_schedule_near_domain (func, data,NUMA_NO_NODE , domain); +} + +static inline async_cookie_t +async_schedule_dev (async_func_t func,struct device *dev)include/linux/async.h struct async_domain { · line 71
Quoted 10 lines
include/linux/async.h +10
-0+{ +return async_schedule_near (func, dev,dev_to_node (dev)); +} + +static inline async_cookie_t +async_schedule_dev_domain (async_func_t func,struct device *dev, +struct async_domain *domain) +{ +return async_schedule_near_domain (func, dev,dev_to_node (dev), domain); +}No kerneldoc for these functions? We lost it for
async_schedule_domain()which used to be documented before this patch :(
Quoted 4 lines
thanks,
greg k-h
I will add it for v3.
Thanks.
- Alex
PATCH v2 3/5 driver core: Probe devices asynchronously instead of the driver
In reply to Alexander DuyckThis change makes it so that we probe devices asynchronously instead of the driver. This results in us seeing the same behavior if the device is registered before the driver or after. This way we can avoid serializing the initialization should the driver not be loaded until after the devices have already been added.
The motivation behind this is that if we have a set of devices that take a significant amount of time to load we can greatly reduce the time to load by processing them in parallel instead of one at a time. In addition, each device can exist on a different node so placing a single thread on one CPU to initialize all of the devices for a given driver can result in poor performance on a system with multiple nodes.
One issue I can see with this patch is that I am using the dev_set/get_drvdata functions to store the driver in the device while I am waiting on the asynchronous init to complete. For now I am protecting it by using the lack of a dev->driver and the device lock.
--- drivers/base/bus.c | 23 +++-------------------- drivers/base/dd.c | 42 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 45 insertions(+), 20 deletions(-)
drivers/base/bus.c +3 -20
@@ -616,17 +616,6 @@ static ssize_t uevent_store(struct device_driver *drv, const char *buf, }static DRIVER_ATTR_WO (uevent); -static void driver_attach_async (void *_drv,async_cookie_t cookie) -{ -struct device_driver *drv = _drv; -int ret; - - ret =driver_attach (drv); - -pr_debug ("bus: '%s': driver %s async attach completed: %d\n" , - drv->bus->name, drv->name, ret); -} -/** * bus_add_driver - Add a driver to the bus. * @drv: driver.
@@ -659,15 +648,9 @@ int bus_add_driver(struct device_driver *drv)klist_add_tail (&priv->knode_bus, &bus->p->klist_drivers);if (drv->bus->p->drivers_autoprobe) { -if (driver_allows_async_probing (drv)) { -pr_debug ("bus: '%s': probing driver %s asynchronously\n" , - drv->bus->name, drv->name); -async_schedule (driver_attach_async, drv); - }else { - error =driver_attach (drv); -if (error) - goto out_unregister; - } + error =driver_attach (drv); +if (error) + goto out_unregister; }module_add_driver (drv->owner, drv);
drivers/base/dd.c +42 -0
54 lines not on this page · raw message
PATCH v2 4/5 driver core: Attach devices on CPU local to device node
In reply to Alexander DuyckThis change makes it so that we call the asynchronous probe routines on a CPU local to the device node. By doing this we should be able to improve our initialization time significantly as we can avoid having to access the device from a remote node which may introduce higher latency.
--- drivers/base/dd.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/base/dd.c +2 -2
@@ -826,7 +826,7 @@ static int __device_attach(struct device *dev, bool allow_async)*/ dev_dbg (dev,"scheduling asynchronous probe\n" );get_device (dev); -async_schedule (__device_attach_async_helper, dev); +async_schedule_dev (__device_attach_async_helper, dev); }else {pm_request_idle (dev); }
@@ -927,7 +927,7 @@ static int __driver_attach(struct device *dev, void *data)if (!dev->driver) {get_device (dev);dev_set_drvdata (dev, drv); -async_schedule (__driver_attach_async_helper, dev); +async_schedule_dev (__driver_attach_async_helper, dev); }device_unlock (dev);return 0 ;
Greg KH
maintainerRe: [workqueue/driver-core PATCH v2 4/5] driver core: Attach devices on CPU local to device node
Quoted 4 lines · Alexander Duyck
This change makes it so that we call the asynchronous probe routines on a CPU local to the device node. By doing this we should be able to improve our initialization time significantly as we can avoid having to access the device from a remote node which may introduce higher latency.
This is nice in theory, but what kind of real numbers does this show? There's a lot of added complexity here, and what is the benifit?
Benchmarks or bootcharts that we can see would be great to have, thanks.
greg k-h
Re: [workqueue/driver-core PATCH v2 4/5] driver core: Attach devices on CPU local to device node
Quoted 6 lines · Greg KH
Quoted 4 lines · Alexander Duyck
This change makes it so that we call the asynchronous probe routines on a CPU local to the device node. By doing this we should be able to improve our initialization time significantly as we can avoid having to access the device from a remote node which may introduce higher latency.
Quoted 7 lines
This is nice in theory, but what kind of real numbers does this show? There's a lot of added complexity here, and what is the benifit?
Benchmarks or bootcharts that we can see would be great to have, thanks.
greg k-h
In the case of persistent memory init the cost for getting the wrong node is pretty significant. On my test system with 3TB per node just getting the initialization node matched up to the memory node dropped initialization time per node from 39 seconds down to about 26 seconds per node.
We are already starting to see code like this pop up in subsystems anyway. For example the PCI code already has logic similar to what I am adding here floating around in it[1]. I'm hoping that by placing this change in the core device code we could start consolidating it so we don't have all the individual drivers or subsystems implementing their own NUMA specific init logic.
This is likely going to become more of an issue in the future as we now have CPUs like the AMD Ryzen Threadripper out there that have people starting to discuss NUMA in the consumer space.
- Alex
[1] https://elixir.bootlin.com/linux/v4.19-rc7/source/drivers/pci/pci-driver.c#L331
PATCH v2 5/5 PM core: Use new async_schedule_dev command
In reply to Alexander DuyckThis change makes it so that we use the device specific version of the async_schedule commands to defer various tasks related to power management. By doing this we should see a slight improvement in performance as any device that is sensitive to latency/locality in the setup will now be initializing on the node closest to the device.
--- drivers/base/power/main.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
drivers/base/power/main.c +6 -6
@@ -726,7 +726,7 @@ void dpm_noirq_resume_devices(pm_message_t state)reinit_completion (&dev->power.completion);if (is_async (dev)) {get_device (dev); -async_schedule (async_resume_noirq, dev); +async_schedule_dev (async_resume_noirq, dev); } }
@@ -883,7 +883,7 @@ void dpm_resume_early(pm_message_t state)reinit_completion (&dev->power.completion);if (is_async (dev)) {get_device (dev); -async_schedule (async_resume_early, dev); +async_schedule_dev (async_resume_early, dev); } }
@@ -1047,7 +1047,7 @@ void dpm_resume(pm_message_t state)reinit_completion (&dev->power.completion);if (is_async (dev)) {get_device (dev); -async_schedule (async_resume, dev); +async_schedule_dev (async_resume, dev); } }
@@ -1366,7 +1366,7 @@ static int device_suspend_noirq(struct device *dev)if (is_async (dev)) {get_device (dev); -async_schedule (async_suspend_noirq, dev); +async_schedule_dev (async_suspend_noirq, dev);return 0 ; }return __device_suspend_noirq (dev, pm_transition, false);
@@ -1569,7 +1569,7 @@ static int device_suspend_late(struct device *dev)if (is_async (dev)) {get_device (dev); -async_schedule (async_suspend_late, dev); +async_schedule_dev (async_suspend_late, dev);return 0 ; }
@@ -1833,7 +1833,7 @@ static int device_suspend(struct device *dev)if (is_async (dev)) {get_device (dev); -async_schedule (async_suspend, dev); +async_schedule_dev (async_suspend, dev);return 0 ; }