Skip to contentSkip to conversation

Add NUMA aware async_schedule calls

PATCH Superseded Alexander Duyck 2 participants 8 messages Oct 11, 2018 – Oct 12, 2018

Subsystems (5)

v3 of 10 Superseded Latest is v10 →

lore
Share & tools
lore
#0 lore raw

v3 cover Add NUMA aware async_schedule calls

This patch set provides functionality that will help to improve the locality of the async_schedule calls used to provide deferred initialization.

This patch set originally started out with me focused on just the one call to async_schedule_domain in the nvdimm tree that was being used to defer the device_add call however after doing some digging I realized the scope of this was much broader than I had originally planned. As such I went through and reworked the underlying infrastructure down to replacing the queue_work call itself with a function of my own and opted to try and provide a NUMA aware solution that would work for a broader audience.

Ideally this solution should be agnostic enough that it can be used for any given device type. So ideally we shouldn't need to implement one-off solutions for each bus as has already kind of happened in the PCI tree with pci_call_probe.

RFC->v1:
Dropped nvdimm patch to submit later.
It relies on code in libnvdimm development tree.
Simplified queue_work_near to just convert node into a CPU. Split up drivers core and PM core patches.
v1->v2:
Renamed queue_work_near to queue_work_node
Added WARN_ON_ONCE if we use queue_work_node with per-cpu workqueue
v2->v3:
Added Acked-by for queue_work_node patch
Continued rename from _near to _node for consistency with queue_work_node
Renamed async_schedule_near_domain to async_schedule_node_domain Renamed async_schedule_near to async_schedule_node
Added kerneldoc for new async_schedule_XXX functions
Added nvdimm boot time improvement data to description of patch 4
---

Alexander Duyck (5):
workqueue: Provide queue_work_node to queue work near a given NUMA node
async: Add support for queueing on specific NUMA node driver core: Probe devices asynchronously instead of the driver driver core: Attach devices on CPU local to device node
PM core: Use new async_schedule_dev command

 drivers/base/bus.c        |   23 ++----------
 drivers/base/dd.c         |   44 +++++++++++++++++++++++-
 drivers/base/power/main.c |   12 +++---
 include/linux/async.h     |   84 +++++++++++++++++++++++++++++++++++++++++++--
 include/linux/workqueue.h |    2 +
 kernel/async.c            |   53 ++++++++++++++++------------
 kernel/workqueue.c        |   84 +++++++++++++++++++++++++++++++++++++++++++++
 7 files changed, 248 insertions(+), 54 deletions(-)

--

#1 lore raw

PATCH v3 1/5 workqueue: Provide queue_work_node to queue work near a given NUMA node

This patch provides a new function queue_work_node which is meant to schedule work on a "random" CPU of the requested NUMA node. The main motivation for this is to help assist asynchronous init to better improve boot times for devices that are local to a specific node.

For now we just default to the first CPU that is in the intersection of the cpumask of the node and the online cpumask. The only exception is if the CPU is local to the node we will just use the current CPU. This should work for our purposes as we are currently only using this for unbound work so the CPU will be translated to a node anyway instead of being directly used.

As we are only using the first CPU to represent the NUMA node for now I am limiting the scope of the function so that it can only be used with unbound workqueues.

---
 include/linux/workqueue.h |    2 +
 kernel/workqueue.c        |   84 +++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 86 insertions(+)
include/linux/workqueue.h +2 -0
index 60d673e15632..1f50c1e586e7 100644
@@ -463,6 +463,8 @@ int apply_workqueue_attrs(struct workqueue_struct *wq,
 
 extern bool queue_work_on(int cpu, struct workqueue_struct *wq,
 			struct work_struct *work);
+extern bool queue_work_node(int node, struct workqueue_struct *wq,
+			    struct work_struct *work);
 extern bool queue_delayed_work_on(int cpu, struct workqueue_struct *wq,
 			struct delayed_work *work, unsigned long delay);
 extern bool mod_delayed_work_on(int cpu, struct workqueue_struct *wq,
kernel/workqueue.c +84 -0

90 lines not on this page · raw message

#2 lore raw

PATCH v3 2/5 async: Add support for queueing on specific NUMA node

In reply to Alexander Duyck

This patch introduces four new variants of the async_schedule_ functions that allow scheduling on a specific NUMA node.

The first two functions are async_schedule_near and async_schedule_near_domain which end up mapping to async_schedule and async_schedule_domain but provide NUMA node specific functionality. They replace the original functions which were moved to inline function definitions that call the new functions while passing NUMA_NO_NODE.

The second two functions are async_schedule_dev and async_schedule_dev_domain which provide NUMA specific functionality when passing a device as the data member and that device has a NUMA node other than NUMA_NO_NODE.

The main motivation behind this is to address the need to be able to schedule device specific init work on specific NUMA nodes in order to improve performance of memory initialization.

---
 include/linux/async.h |   84 +++++++++++++++++++++++++++++++++++++++++++++++--
 kernel/async.c        |   53 +++++++++++++++++--------------
 2 files changed, 110 insertions(+), 27 deletions(-)
include/linux/async.h +81 -3
index 6b0226bdaadc..98a94e6e367d 100644
@@ -14,6 +14,8 @@
 
 #include <linux/types.h>
 #include <linux/list.h>
+#include <linux/numa.h>
+#include <linux/device.h>
 
 typedef u64 async_cookie_t;
 typedef void (*async_func_t) (void *data, async_cookie_t cookie);
@@ -37,9 +39,85 @@ struct async_domain {
 	struct async_domain _name = { .pending = LIST_HEAD_INIT(_name.pending), \
 				      .registered = 0 }
 
-extern async_cookie_t async_schedule(async_func_t func, void *data);
-extern async_cookie_t async_schedule_domain(async_func_t func, void *data,
-					    struct async_domain *domain);
+async_cookie_t async_schedule_node(async_func_t func, void *data,
+				   int node);
+async_cookie_t async_schedule_node_domain(async_func_t func, void *data,
+					  int node,
+					  struct async_domain *domain);
+
+/**
+ * async_schedule - schedule a function for asynchronous execution
+ * @func: function to execute asynchronously
+ * @data: data pointer to pass to the function
+ *
+ * Returns an async_cookie_t that may be used for checkpointing later.
+ * Note: This function may be called from atomic or non-atomic contexts.
+ */
+static inline async_cookie_t async_schedule(async_func_t func, void *data)
+{
+	return async_schedule_node(func, data, NUMA_NO_NODE);
+}
+
+/**
+ * async_schedule_domain - schedule a function for asynchronous execution within a certain domain
+ * @func: function to execute asynchronously
+ * @data: data pointer to pass to the function
+ * @domain: the domain
+ *
+ * Returns an async_cookie_t that may be used for checkpointing later.
+ * @domain may be used in the async_synchronize_*_domain() functions to
+ * wait within a certain synchronization domain rather than globally.  A
+ * synchronization domain is specified via @domain.
+ * Note: This function may be called from atomic or non-atomic contexts.
+ */
+static inline async_cookie_t
+async_schedule_domain(async_func_t func, void *data,
+		      struct async_domain *domain)
+{
+	return async_schedule_node_domain(func, data, NUMA_NO_NODE, domain);
+}
+
+/**
+ * async_schedule_dev - A device specific version of async_schedule
+ * @func: function to execute asynchronously
+ * @dev: device argument to be passed to function
+ *
+ * Returns an async_cookie_t that may be used for checkpointing later.
+ * @dev is used as both the argument for the function and to provide NUMA
+ * context for where to run the function. By doing this we can try to
+ * provide for the best possible outcome by operating on the device on the
+ * CPUs closest to the device.
+ * Note: This function may be called from atomic or non-atomic contexts.
+ */
+static inline async_cookie_t
+async_schedule_dev(async_func_t func, struct device *dev)
+{
+	return async_schedule_node(func, dev, dev_to_node(dev));
+}
+
+/**
+ * async_schedule_dev_domain - A device specific version of async_schedule_domain
+ * @func: function to execute asynchronously
+ * @dev: device argument to be passed to function
+ * @domain: the domain
+ *
+ * Returns an async_cookie_t that may be used for checkpointing later.
+ * @dev is used as both the argument for the function and to provide NUMA
+ * context for where to run the function. By doing this we can try to
+ * provide for the best possible outcome by operating on the device on the
+ * CPUs closest to the device.
+ * @domain may be used in the async_synchronize_*_domain() functions to
+ * wait within a certain synchronization domain rather than globally.  A
+ * synchronization domain is specified via @domain.
+ * Note: This function may be called from atomic or non-atomic contexts.
+ */
+static inline async_cookie_t
+async_schedule_dev_domain(async_func_t func, struct device *dev,
+			  struct async_domain *domain)
+{
+	return async_schedule_node_domain(func, dev, dev_to_node(dev), domain);
+}
+
 void async_unregister_domain(struct async_domain *domain);
 extern void async_synchronize_full(void);
 extern void async_synchronize_full_domain(struct async_domain *domain);
kernel/async.c +29 -24

79 lines not on this page · raw message

#3 lore raw

PATCH v3 3/5 driver core: Probe devices asynchronously instead of the driver

In reply to Alexander Duyck

This change makes it so that we probe devices asynchronously instead of the driver. This results in us seeing the same behavior if the device is registered before the driver or after. This way we can avoid serializing the initialization should the driver not be loaded until after the devices have already been added.

The motivation behind this is that if we have a set of devices that take a significant amount of time to load we can greatly reduce the time to load by processing them in parallel instead of one at a time. In addition, each device can exist on a different node so placing a single thread on one CPU to initialize all of the devices for a given driver can result in poor performance on a system with multiple nodes.

One issue I can see with this patch is that I am using the dev_set/get_drvdata functions to store the driver in the device while I am waiting on the asynchronous init to complete. For now I am protecting it by using the lack of a dev->driver and the device lock.

---
 drivers/base/bus.c |   23 +++--------------------
 drivers/base/dd.c  |   42 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 45 insertions(+), 20 deletions(-)
drivers/base/bus.c +3 -20
index 8bfd27ec73d6..2a17bed657ec 100644
@@ -616,17 +616,6 @@ static ssize_t uevent_store(struct device_driver *drv, const char *buf,
 }
 static DRIVER_ATTR_WO(uevent);
 
-static void driver_attach_async(void *_drv, async_cookie_t cookie)
-{
-	struct device_driver *drv = _drv;
-	int ret;
-
-	ret = driver_attach(drv);
-
-	pr_debug("bus: '%s': driver %s async attach completed: %d\n",
-		 drv->bus->name, drv->name, ret);
-}
-
 /**
  * bus_add_driver - Add a driver to the bus.
  * @drv: driver.
@@ -659,15 +648,9 @@ int bus_add_driver(struct device_driver *drv)
 
 	klist_add_tail(&priv->knode_bus, &bus->p->klist_drivers);
 	if (drv->bus->p->drivers_autoprobe) {
-		if (driver_allows_async_probing(drv)) {
-			pr_debug("bus: '%s': probing driver %s asynchronously\n",
-				drv->bus->name, drv->name);
-			async_schedule(driver_attach_async, drv);
-		} else {
-			error = driver_attach(drv);
-			if (error)
-				goto out_unregister;
-		}
+		error = driver_attach(drv);
+		if (error)
+			goto out_unregister;
 	}
 	module_add_driver(drv->owner, drv);
 
drivers/base/dd.c +42 -0

54 lines not on this page · raw message

#4 lore raw

Re: [driver-core PATCH v3 3/5] driver core: Probe devices asynchronously instead of the driver

Quoted 83 lines · Alexander Duyck

This change makes it so that we probe devices asynchronously instead of the driver. This results in us seeing the same behavior if the device is registered before the driver or after. This way we can avoid serializing the initialization should the driver not be loaded until after the devices have already been added.

The motivation behind this is that if we have a set of devices that take a significant amount of time to load we can greatly reduce the time to load by processing them in parallel instead of one at a time. In addition, each device can exist on a different node so placing a single thread on one CPU to initialize all of the devices for a given driver can result in poor performance on a system with multiple nodes.

One issue I can see with this patch is that I am using the dev_set/get_drvdata functions to store the driver in the device while I am waiting on the asynchronous init to complete. For now I am protecting it by using the lack of a dev->driver and the device lock.

---
 drivers/base/bus.c |   23 +++--------------------
 drivers/base/dd.c  |   42 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 45 insertions(+), 20 deletions(-)
drivers/base/bus.c +3 -20
index 8bfd27ec73d6..2a17bed657ec 100644
@@ -616,17 +616,6 @@ static ssize_t uevent_store(struct device_driver *drv, const char *buf,
 }
 static DRIVER_ATTR_WO(uevent);
 
-static void driver_attach_async(void *_drv, async_cookie_t cookie)
-{
-       struct device_driver *drv = _drv;
-       int ret;
-
-       ret = driver_attach(drv);
-
-       pr_debug("bus: '%s': driver %s async attach completed: %d\n",
-                drv->bus->name, drv->name, ret);
-}
-
 /**
  * bus_add_driver - Add a driver to the bus.
  * @drv: driver.
@@ -659,15 +648,9 @@ int bus_add_driver(struct device_driver *drv)
 
        klist_add_tail(&priv->knode_bus, &bus->p->klist_drivers);
        if (drv->bus->p->drivers_autoprobe) {
-               if (driver_allows_async_probing(drv)) {
-                       pr_debug("bus: '%s': probing driver %s asynchronously\n",
-                               drv->bus->name, drv->name);
-                       async_schedule(driver_attach_async, drv);
-               } else {
-                       error = driver_attach(drv);
-                       if (error)
-                               goto out_unregister;
-               }
+               error = driver_attach(drv);
+               if (error)
+                       goto out_unregister;
        }
        module_add_driver(drv->owner, drv);
 
drivers/base/dd.c +9 -0
index 169412ee4ae8..5ba366c1cb83 100644
@@ -864,6 +864,29 @@ void device_initial_probe(struct device *dev)
        __device_attach(dev, true);
 }
 
+static void __driver_attach_async_helper(void *_dev, async_cookie_t cookie)
+{
+       struct device *dev = _dev;
+
+       if (dev->parent && dev->bus->need_parent_lock)
+               device_lock(dev->parent);
+       device_lock(dev);
+
+       if (!dev->driver) {

drivers/base/dd.c void device_initial_probe(struct device *dev) · line 885

Quoted 10 lines
drivers/base/dd.c +10 -0
+               struct device_driver *drv = dev_get_drvdata(dev);
+
+               driver_probe_device(drv, dev);
+       }
+
+       dev_dbg(dev, "async probe completed\n");
+
+       device_unlock(dev);
+       if (dev->parent && dev->bus->need_parent_lock)
+               device_unlock(dev->parent);

The above duplicates some code from __driver_attach().

You could avoid that by adding one more function to call directly from __driver_attach() in the "sync" case and from here, along the lines of what the PM code does.

Quoted 34 lines
drivers/base/dd.c +23 -0
+
+       put_device(dev);
+}
+
 static int __driver_attach(struct device *dev, void *data)
 {
        struct device_driver *drv = data;
@@ -891,6 +914,25 @@ static int __driver_attach(struct device *dev, void *data)
                return ret;
        } /* ret > 0 means positive match */
 
+       if (driver_allows_async_probing(drv)) {
+               /*
+                * Instead of probing the device synchronously we will
+                * probe it asynchronously to allow for more parallelism.
+                *
+                * We only take the device lock here in order to guarantee
+                * that the dev->driver and driver_data fields are protected
+                */
+               dev_dbg(dev, "scheduling asynchronous probe\n");
+               device_lock(dev);
+               if (!dev->driver) {
+                       get_device(dev);
+                       dev_set_drvdata(dev, drv);
+                       async_schedule(__driver_attach_async_helper, dev);
+               }
+               device_unlock(dev);
+               return 0;
+       }
+
        if (dev->parent && dev->bus->need_parent_lock)
                device_lock(dev->parent);
        device_lock(dev);
#5 lore raw

PATCH v3 4/5 driver core: Attach devices on CPU local to device node

In reply to Alexander Duyck

This change makes it so that we call the asynchronous probe routines on a CPU local to the device node. By doing this we should be able to improve our initialization time significantly as we can avoid having to access the device from a remote node which may introduce higher latency.

For example, in the case of initializing memory for NVDIMM this can have a singifcant impact as initialing 3TB on remote node can take up to 39 seconds while initialing it on a local node only takes 23 seconds. It is situations like this where we will see the biggest improvement.

---
 drivers/base/dd.c |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/base/dd.c +2 -2
index 5ba366c1cb83..81472dc44a70 100644
@@ -826,7 +826,7 @@ static int __device_attach(struct device *dev, bool allow_async)
 			 */
 			dev_dbg(dev, "scheduling asynchronous probe\n");
 			get_device(dev);
-			async_schedule(__device_attach_async_helper, dev);
+			async_schedule_dev(__device_attach_async_helper, dev);
 		} else {
 			pm_request_idle(dev);
 		}
@@ -927,7 +927,7 @@ static int __driver_attach(struct device *dev, void *data)
 		if (!dev->driver) {
 			get_device(dev);
 			dev_set_drvdata(dev, drv);
-			async_schedule(__driver_attach_async_helper, dev);
+			async_schedule_dev(__driver_attach_async_helper, dev);
 		}
 		device_unlock(dev);
 		return 0;
#6 lore raw

PATCH v3 5/5 PM core: Use new async_schedule_dev command

In reply to Alexander Duyck

This change makes it so that we use the device specific version of the async_schedule commands to defer various tasks related to power management. By doing this we should see a slight improvement in performance as any device that is sensitive to latency/locality in the setup will now be initializing on the node closest to the device.

---
 drivers/base/power/main.c |   12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)
drivers/base/power/main.c +6 -6
index a690fd400260..ebb8b61b52e9 100644
@@ -726,7 +726,7 @@ void dpm_noirq_resume_devices(pm_message_t state)
 		reinit_completion(&dev->power.completion);
 		if (is_async(dev)) {
 			get_device(dev);
-			async_schedule(async_resume_noirq, dev);
+			async_schedule_dev(async_resume_noirq, dev);
 		}
 	}
 
@@ -883,7 +883,7 @@ void dpm_resume_early(pm_message_t state)
 		reinit_completion(&dev->power.completion);
 		if (is_async(dev)) {
 			get_device(dev);
-			async_schedule(async_resume_early, dev);
+			async_schedule_dev(async_resume_early, dev);
 		}
 	}
 
@@ -1047,7 +1047,7 @@ void dpm_resume(pm_message_t state)
 		reinit_completion(&dev->power.completion);
 		if (is_async(dev)) {
 			get_device(dev);
-			async_schedule(async_resume, dev);
+			async_schedule_dev(async_resume, dev);
 		}
 	}
 
@@ -1366,7 +1366,7 @@ static int device_suspend_noirq(struct device *dev)
 
 	if (is_async(dev)) {
 		get_device(dev);
-		async_schedule(async_suspend_noirq, dev);
+		async_schedule_dev(async_suspend_noirq, dev);
 		return 0;
 	}
 	return __device_suspend_noirq(dev, pm_transition, false);
@@ -1569,7 +1569,7 @@ static int device_suspend_late(struct device *dev)
 
 	if (is_async(dev)) {
 		get_device(dev);
-		async_schedule(async_suspend_late, dev);
+		async_schedule_dev(async_suspend_late, dev);
 		return 0;
 	}
 
@@ -1833,7 +1833,7 @@ static int device_suspend(struct device *dev)
 
 	if (is_async(dev)) {
 		get_device(dev);
-		async_schedule(async_suspend, dev);
+		async_schedule_dev(async_suspend, dev);
 		return 0;
 	}
 
maintainer Reviewed-by on v3 5/5
#7 lore raw

Re: [driver-core PATCH v3 5/5] PM core: Use new async_schedule_dev command

Quoted 8 lines · Alexander Duyck

This change makes it so that we use the device specific version of the async_schedule commands to defer various tasks related to power management. By doing this we should see a slight improvement in performance as any device that is sensitive to latency/locality in the setup will now be initializing on the node closest to the device.

Quoted 63 lines
---
 drivers/base/power/main.c |   12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)
drivers/base/power/main.c +6 -6
index a690fd400260..ebb8b61b52e9 100644
@@ -726,7 +726,7 @@ void dpm_noirq_resume_devices(pm_message_t state)
                reinit_completion(&dev->power.completion);
                if (is_async(dev)) {
                        get_device(dev);
-                       async_schedule(async_resume_noirq, dev);
+                       async_schedule_dev(async_resume_noirq, dev);
                }
        }
 
@@ -883,7 +883,7 @@ void dpm_resume_early(pm_message_t state)
                reinit_completion(&dev->power.completion);
                if (is_async(dev)) {
                        get_device(dev);
-                       async_schedule(async_resume_early, dev);
+                       async_schedule_dev(async_resume_early, dev);
                }
        }
 
@@ -1047,7 +1047,7 @@ void dpm_resume(pm_message_t state)
                reinit_completion(&dev->power.completion);
                if (is_async(dev)) {
                        get_device(dev);
-                       async_schedule(async_resume, dev);
+                       async_schedule_dev(async_resume, dev);
                }
        }
 
@@ -1366,7 +1366,7 @@ static int device_suspend_noirq(struct device *dev)
 
        if (is_async(dev)) {
                get_device(dev);
-               async_schedule(async_suspend_noirq, dev);
+               async_schedule_dev(async_suspend_noirq, dev);
                return 0;
        }
        return __device_suspend_noirq(dev, pm_transition, false);
@@ -1569,7 +1569,7 @@ static int device_suspend_late(struct device *dev)
 
        if (is_async(dev)) {
                get_device(dev);
-               async_schedule(async_suspend_late, dev);
+               async_schedule_dev(async_suspend_late, dev);
                return 0;
        }
 
@@ -1833,7 +1833,7 @@ static int device_suspend(struct device *dev)
 
        if (is_async(dev)) {
                get_device(dev);
-               async_schedule(async_suspend, dev);
+               async_schedule_dev(async_suspend, dev);
                return 0;
        }