Skip to contentSkip to conversation

Add NUMA aware async_schedule calls

RFC PATCH In review Alexander Duyck 3 participants 25 messages Sep 26, 2018 – Oct 2, 2018

Subsystems (6)
lore
Share & tools
lore
#0 lore raw

cover Add NUMA aware async_schedule calls

This patch set provides functionality that will help to improve the locality of the async_schedule calls used to provide deferred initialization.

This patch set originally started out with me focused on just the one call to async_schedule_domain in the nvdimm tree that was being used to defer the device_add call however after doing some digging I realized the scope of this was much broader than I had originally planned. As such I went through and reworked the underlying infrastructure down to replacing the queue_work call itself with a function of my own and opted to try and provide a NUMA aware solution that would work for a broader audience.

I am submitting this an RFC to figure out just how far off I am from where I need to be on this patch set and to determine which tree I should ultimately be submitting this to.

---

Alexander Duyck (5):
workqueue: Provide queue_work_near to queue work near a given NUMA node
async: Add support for queueing on specific NUMA node driver core: Probe devices asynchronously instead of the driver driver core: Use new async_schedule_dev command
nvdimm: Schedule device registration on node local to the device

 drivers/base/bus.c        |   23 +-------
 drivers/base/dd.c         |   44 +++++++++++++++
 drivers/base/power/main.c |   12 ++--
 drivers/nvdimm/bus.c      |   11 +++-
 include/linux/async.h     |   27 ++++++++-
 include/linux/workqueue.h |    2 +
 kernel/async.c            |   62 ++++++++++++++++------
 kernel/workqueue.c        |  129 ++++++++++++++++++++++++++++++++++++++++++++-
 8 files changed, 258 insertions(+), 52 deletions(-)

--

#1 lore raw

RFC PATCH 1/5 workqueue: Provide queue_work_near to queue work near a given NUMA node

This patch provides a new function queue_work_near which is meant to schedule work on the nearest unbound CPU to the requested NUMA node. The main motivation for this is to help assist asynchronous init to better improve boot times for devices that are local to a specific node.

---
 include/linux/workqueue.h |    2 +
 kernel/workqueue.c        |  129 ++++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 129 insertions(+), 2 deletions(-)
include/linux/workqueue.h +2 -0
index 60d673e15632..1f9f0a65437b 100644
@@ -463,6 +463,8 @@ int apply_workqueue_attrs(struct workqueue_struct *wq,
 
 extern bool queue_work_on(int cpu, struct workqueue_struct *wq,
 			struct work_struct *work);
+extern bool queue_work_near(int node, struct workqueue_struct *wq,
+			    struct work_struct *work);
 extern bool queue_delayed_work_on(int cpu, struct workqueue_struct *wq,
 			struct delayed_work *work, unsigned long delay);
 extern bool mod_delayed_work_on(int cpu, struct workqueue_struct *wq,
kernel/workqueue.c +127 -2

154 lines not on this page · raw message

maintainer
#2 lore raw

Re: [RFC workqueue/driver-core PATCH 1/5] workqueue: Provide queue_work_near to queue work near a given NUMA node

Hello,

Quoted 6 lines · Alexander Duyck

This patch provides a new function queue_work_near which is meant to schedule work on the nearest unbound CPU to the requested NUMA node. The main motivation for this is to help assist asynchronous init to better improve boot times for devices that are local to a specific node.

Why not just use unbound workqueues, which are NUMA-affine by default? Are there big enough advantages?

Thanks.

Signature · 1 line
tejun
#3 lore raw

Re: [RFC workqueue/driver-core PATCH 1/5] workqueue: Provide queue_work_near to queue work near a given NUMA node

Quoted 8 lines · Tejun Heo

Hello,

Quoted 5 lines · Alexander Duyck

This patch provides a new function queue_work_near which is meant to schedule work on the nearest unbound CPU to the requested NUMA node. The main motivation for this is to help assist asynchronous init to better improve boot times for devices that are local to a specific node.

Quoted 6 lines
Quoted 1 line

Why not just use unbound workqueues, which are NUMA-affine by default? Are there big enough advantages?

Thanks.

I am using unbound workqueues. However there isn't an interface that exposes the NUMA bits of them directly. All I am doing with this patch is adding "queue_work_near" which takes a NUMA node as an argument and then copies the logic of "queue_work_on" with the exception that I am doing a check to verify that there is an intersection between wq_unbound_cpumask and the cpumask of the node, and then passing a CPU from that intersection into "__queue_work".

Thanks.

- Alex

maintainer
#4 lore raw

Re: [RFC workqueue/driver-core PATCH 1/5] workqueue: Provide queue_work_near to queue work near a given NUMA node

Hello,

Quoted 7 lines · Alexander Duyck

I am using unbound workqueues. However there isn't an interface that exposes the NUMA bits of them directly. All I am doing with this patch is adding "queue_work_near" which takes a NUMA node as an argument and then copies the logic of "queue_work_on" with the exception that I am doing a check to verify that there is an intersection between wq_unbound_cpumask and the cpumask of the node, and then passing a CPU from that intersection into "__queue_work".

Can it just take a cpu id and not feed that to __queue_work()? That looks like a lot of extra logic.

Thanks.

Signature · 1 line
tejun
#5 lore raw

Re: [RFC workqueue/driver-core PATCH 1/5] workqueue: Provide queue_work_near to queue work near a given NUMA node

Quoted 11 lines · Tejun Heo

Hello,

Quoted 7 lines · Alexander Duyck

I am using unbound workqueues. However there isn't an interface that exposes the NUMA bits of them directly. All I am doing with this patch is adding "queue_work_near" which takes a NUMA node as an argument and then copies the logic of "queue_work_on" with the exception that I am doing a check to verify that there is an intersection between wq_unbound_cpumask and the cpumask of the node, and then passing a CPU from that intersection into "__queue_work".

Quoted 4 lines

Can it just take a cpu id and not feed that to __queue_work()? That looks like a lot of extra logic.

Thanks.

I could just use queue_work_on probably, but is there any issue if I am passing CPU values that are not in the wq_unbound_cpumask? That was mostly my concern. Also for an unbound queue do I need to worry about the hotplug lock? I wasn't sure if that was the case or not as I know it is called out as something to be concerned with using queue_work_on, but in __queue_work the value is just used to determine which node to grab a work queue from.

I forgot to address your question about the advantages. They are pretty significant. The test system I was working with was initializing 3TB of nvdimm memory per node. If the node is aligned it takes something like 24 seconds, whereas an unaligned core can take 36 seconds or more.

Thanks.

- Alex

maintainer
#6 lore raw

Re: [RFC workqueue/driver-core PATCH 1/5] workqueue: Provide queue_work_near to queue work near a given NUMA node

Hello,

Quoted 3 lines · Alexander Duyck

On 9/26/2018 3:09 PM, Tejun Heo wrote:
I could just use queue_work_on probably, but is there any issue if I am passing CPU values that are not in the wq_unbound_cpumask? That

That should be fine. If it can't find any available cpu, it'll fall back to round-robin. We probably can improve it so that it can consider the numa distance when falling back.

Quoted 2 lines

was mostly my concern. Also for an unbound queue do I need to worry about the hotplug lock? I wasn't sure if that was the case or not as

Issuers don't need to worry about them.

Quoted 3 lines

I know it is called out as something to be concerned with using queue_work_on, but in __queue_work the value is just used to determine which node to grab a work queue from.

It might be better to leave queue_work_on() to be used for per-cpu workqueues and introduce queue_work_near() as you suggseted. I just don't want it to duplicate the node selection code in it. Would that work?

Quoted 5 lines

I forgot to address your question about the advantages. They are pretty significant. The test system I was working with was initializing 3TB of nvdimm memory per node. If the node is aligned it takes something like 24 seconds, whereas an unaligned core can take 36 seconds or more.

Oh yeah, sure, numa affinity matters quite a bit on memory heavy workloads. I was mistaken that you were adding adding numa affinity to per-cpu workqueues.

Thanks.

Signature · 1 line
tejun
#7 lore raw

Re: [RFC workqueue/driver-core PATCH 1/5] workqueue: Provide queue_work_near to queue work near a given NUMA node

Quoted 16 lines · Tejun Heo

Hello,

Quoted 3 lines · Alexander Duyck

On 9/26/2018 3:09 PM, Tejun Heo wrote:
I could just use queue_work_on probably, but is there any issue if I am passing CPU values that are not in the wq_unbound_cpumask? That

That should be fine. If it can't find any available cpu, it'll fall back to round-robin. We probably can improve it so that it can consider the numa distance when falling back.

Quoted 2 lines

was mostly my concern. Also for an unbound queue do I need to worry about the hotplug lock? I wasn't sure if that was the case or not as

Issuers don't need to worry about them.

Quoted 8 lines
Quoted 3 lines

I know it is called out as something to be concerned with using queue_work_on, but in __queue_work the value is just used to determine which node to grab a work queue from.

It might be better to leave queue_work_on() to be used for per-cpu workqueues and introduce queue_work_near() as you suggseted. I just don't want it to duplicate the node selection code in it. Would that work?

So if I understand what you are saying correctly we default to round-robin on a given node has no CPUs attached to it. I could probably work with that if that is the default behavior instead of adding much of the complexity I already have.

The question I have then is what should I do about workqueues that aren't WQ_UNBOUND if they attempt to use queue_work_near? In that case I should be looking for some way to go from a node to a CPU shouldn't I? If so should I look at doing something like wq_select_unbound_cpu that uses the node cpumask instead of the wq_unbound_cpumask?

- Alex

maintainer
#8 lore raw

Re: [RFC workqueue/driver-core PATCH 1/5] workqueue: Provide queue_work_near to queue work near a given NUMA node

Hello,

Quoted 9 lines · Alexander Duyck
Quoted 4 lines

It might be better to leave queue_work_on() to be used for per-cpu workqueues and introduce queue_work_near() as you suggseted. I just don't want it to duplicate the node selection code in it. Would that work?

So if I understand what you are saying correctly we default to round-robin on a given node has no CPUs attached to it. I could probably work with that if that is the default behavior instead of adding much of the complexity I already have.

Yeah, it's all in wq_select_unbound_cpu(). Right now, if the requested cpu isn't in wq_unbound_cpumask, it falls back to dumb round-robin. We can probably do better there and find the nearest node considering topology.

Quoted 2 lines

The question I have then is what should I do about workqueues that aren't WQ_UNBOUND if they attempt to use queue_work_near? In that

Hmm... yeah, let's just use queue_work_on() for now. We can sort it out later and users could already do that anyway.

Thanks.

Signature · 1 line
tejun
#9 lore raw

Re: [RFC workqueue/driver-core PATCH 1/5] workqueue: Provide queue_work_near to queue work near a given NUMA node

Quoted 13 lines · Tejun Heo

Hello,

Quoted 9 lines · Alexander Duyck
Quoted 4 lines

It might be better to leave queue_work_on() to be used for per-cpu workqueues and introduce queue_work_near() as you suggseted. I just don't want it to duplicate the node selection code in it. Would that work?

So if I understand what you are saying correctly we default to round-robin on a given node has no CPUs attached to it. I could probably work with that if that is the default behavior instead of adding much of the complexity I already have.

Quoted 4 lines

Yeah, it's all in wq_select_unbound_cpu(). Right now, if the requested cpu isn't in wq_unbound_cpumask, it falls back to dumb round-robin. We can probably do better there and find the nearest node considering topology.

Well if we could get wq_select_unbound_cpu doing the right thing based on node topology that would be most of my work solved right there. Basically I could just pass WQ_CPU_UNBOUND with the correct node and it would take care of getting to the right CPU.

Quoted 7 lines
Quoted 2 lines

The question I have then is what should I do about workqueues that aren't WQ_UNBOUND if they attempt to use queue_work_near? In that

Hmm... yeah, let's just use queue_work_on() for now. We can sort it out later and users could already do that anyway.

Thanks.

So are you saying I should just return an error for now if somebody tries to use something other than an unbound workqueue with queue_work_near, and expect everyone else to just use queue_work_on for the other workqueue types?

Thanks.

- Alex

maintainer
#10 lore raw

Re: [RFC workqueue/driver-core PATCH 1/5] workqueue: Provide queue_work_near to queue work near a given NUMA node

Hello,

Quoted 9 lines · Alexander Duyck
Quoted 4 lines

Yeah, it's all in wq_select_unbound_cpu(). Right now, if the requested cpu isn't in wq_unbound_cpumask, it falls back to dumb round-robin. We can probably do better there and find the nearest node considering topology.

Well if we could get wq_select_unbound_cpu doing the right thing based on node topology that would be most of my work solved right there. Basically I could just pass WQ_CPU_UNBOUND with the correct node and it would take care of getting to the right CPU.

Yeah, sth like that. It might be better to keep the function to take cpu for consistency as everything else passes around cpu.

Quoted 10 lines
Quoted 5 lines
Quoted 2 lines

The question I have then is what should I do about workqueues that aren't WQ_UNBOUND if they attempt to use queue_work_near? In that

Hmm... yeah, let's just use queue_work_on() for now. We can sort it out later and users could already do that anyway.

So are you saying I should just return an error for now if somebody tries to use something other than an unbound workqueue with queue_work_near, and expect everyone else to just use queue_work_on for the other workqueue types?

Oh, I meant that let's not add a new interface for now and just use queue_work_on() for your use case too.

Thanks.

Signature · 1 line
tejun
#11 lore raw

Re: [RFC workqueue/driver-core PATCH 1/5] workqueue: Provide queue_work_near to queue work near a given NUMA node

Quoted 27 lines · Tejun Heo

Hello,

Quoted 9 lines · Alexander Duyck
Quoted 4 lines

Yeah, it's all in wq_select_unbound_cpu(). Right now, if the requested cpu isn't in wq_unbound_cpumask, it falls back to dumb round-robin. We can probably do better there and find the nearest node considering topology.

Well if we could get wq_select_unbound_cpu doing the right thing based on node topology that would be most of my work solved right there. Basically I could just pass WQ_CPU_UNBOUND with the correct node and it would take care of getting to the right CPU.

Yeah, sth like that. It might be better to keep the function to take cpu for consistency as everything else passes around cpu.

Quoted 10 lines
Quoted 5 lines
Quoted 2 lines

The question I have then is what should I do about workqueues that aren't WQ_UNBOUND if they attempt to use queue_work_near? In that

Hmm... yeah, let's just use queue_work_on() for now. We can sort it out later and users could already do that anyway.

So are you saying I should just return an error for now if somebody tries to use something other than an unbound workqueue with queue_work_near, and expect everyone else to just use queue_work_on for the other workqueue types?

Quoted 4 lines

Oh, I meant that let's not add a new interface for now and just use queue_work_on() for your use case too.

Thanks.

So the only issue is that I was hoping to get away with not having to add additional preemption. That was the motivation behind doing queue_work_near as I could just wrap it all in the same local_irq_save that way I don't have to worry about the CPU I am on changing.

What I may look at doing is just greatly reducing the workqueue_select_unbound_cpu_near function to essentially just perform a few tests and then will just use the results from a cpumask_any_and of the cpumask_of_node and the cpu_online_mask. I'll probably rename it while I am at it since I am going to probably be getting away from the "unbound" checks in the logic.

- Alex

_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm

#12 lore raw

RFC PATCH 2/5 async: Add support for queueing on specific NUMA node

In reply to Alexander Duyck

This patch introduces four new variants of the async_schedule_ functions that allow scheduling on a specific NUMA node.

The first two functions are async_schedule_near and async_schedule_near_domain which end up mapping to async_schedule and async_schedule_domain but provide NUMA node specific functionality. They replace the original functions which were moved to inline function definitions that call the new functions while passing NUMA_NO_NODE.

The second two functions are async_schedule_dev and async_schedule_dev_domain which provide NUMA specific functionality when passing a device as the data member and that device has a NUMA node other than NUMA_NO_NODE.

The main motivation behind this is to address the need to be able to schedule device specific init work on specific NUMA nodes in order to improve performance of memory initialization.

---
 include/linux/async.h |   27 +++++++++++++++++++--
 kernel/async.c        |   62 ++++++++++++++++++++++++++++++++++++-------------
 2 files changed, 69 insertions(+), 20 deletions(-)
include/linux/async.h +24 -3
index 6b0226bdaadc..abf3ee9102df 100644
@@ -14,6 +14,9 @@
 
 #include <linux/types.h>
 #include <linux/list.h>
+#include <linux/numa.h>
+
+struct device;
 
 typedef u64 async_cookie_t;
 typedef void (*async_func_t) (void *data, async_cookie_t cookie);
@@ -37,9 +40,27 @@ struct async_domain {
 	struct async_domain _name = { .pending = LIST_HEAD_INIT(_name.pending), \
 				      .registered = 0 }
 
-extern async_cookie_t async_schedule(async_func_t func, void *data);
-extern async_cookie_t async_schedule_domain(async_func_t func, void *data,
-					    struct async_domain *domain);
+async_cookie_t async_schedule_near(async_func_t func, void *data,
+				   int node);
+async_cookie_t async_schedule_near_domain(async_func_t func, void *data,
+					  int node,
+					  struct async_domain *domain);
+
+static inline async_cookie_t async_schedule(async_func_t func, void *data)
+{
+	return async_schedule_near(func, data, NUMA_NO_NODE);
+}
+
+static inline async_cookie_t
+async_schedule_domain(async_func_t func, void *data,
+		      struct async_domain *domain)
+{
+	return async_schedule_near_domain(func, data, NUMA_NO_NODE, domain);
+}
+
+async_cookie_t async_schedule_dev(async_func_t func, struct device *dev);
+async_cookie_t async_schedule_dev_domain(async_func_t func, struct device *dev,
+					 struct async_domain *domain);
 void async_unregister_domain(struct async_domain *domain);
 extern void async_synchronize_full(void);
 extern void async_synchronize_full_domain(struct async_domain *domain);
kernel/async.c +45 -17

101 lines not on this page · raw message

#13 lore raw

Re: [RFC workqueue/driver-core PATCH 2/5] async: Add support for queueing on specific NUMA node

Quoted 20 lines · Alexander Duyck

This patch introduces four new variants of the async_schedule_ functions that allow scheduling on a specific NUMA node.

The first two functions are async_schedule_near and async_schedule_near_domain which end up mapping to async_schedule and async_schedule_domain but provide NUMA node specific functionality. They replace the original functions which were moved to inline function definitions that call the new functions while passing NUMA_NO_NODE.

The second two functions are async_schedule_dev and async_schedule_dev_domain which provide NUMA specific functionality when passing a device as the data member and that device has a NUMA node other than NUMA_NO_NODE.

The main motivation behind this is to address the need to be able to schedule device specific init work on specific NUMA nodes in order to improve performance of memory initialization.

[..]

Quoted 10 lines

/**
- * async_schedule - schedule a function for asynchronous execution
+ * async_schedule_near - schedule a function for asynchronous execution
* @func: function to execute asynchronously
* @data: data pointer to pass to the function
+ * @node: NUMA node that we want to schedule this on or close to *
* Returns an async_cookie_t that may be used for checkpointing later.
* Note: This function may be called from atomic or non-atomic contexts. */

kernel/async.c static async_cookie_t __async_schedule(async_func_t func, void *data, struct asy · line 232

Quoted 8 lines
kernel/async.c +3 -3
-async_cookie_t async_schedule(async_func_t func, void *data)
+async_cookie_t async_schedule_near(async_func_t func, void *data, int node)
 {
-       return __async_schedule(func, data, &async_dfl_domain);
+       return async_schedule_near_domain(func, data, node, &async_dfl_domain);
 }
-EXPORT_SYMBOL_GPL(async_schedule);
+EXPORT_SYMBOL_GPL(async_schedule_near);

Looks good to me. The _near() suffix makes it clear that we're doing a best effort hint to the work placement compared to the strict expectations of _on routines.

Quoted 17 lines

/**
- * async_schedule_domain - schedule a function for asynchronous execution within a certain domain
+ * async_schedule_dev_domain - schedule a function for asynchronous execution within a certain domain
* @func: function to execute asynchronously
- * @data: data pointer to pass to the function
+ * @dev: device that we are scheduling this work for
* @domain: the domain
*
- * Returns an async_cookie_t that may be used for checkpointing later.
- * @domain may be used in the async_synchronize_*_domain() functions to
- * wait within a certain synchronization domain rather than globally. A
- * synchronization domain is specified via @domain. Note: This function
- * may be called from atomic or non-atomic contexts.
+ * Device specific version of async_schedule_near_domain that provides some
+ * NUMA awareness based on the device node.
+ */

kernel/async.c static async_cookie_t __async_schedule(async_func_t func, void *data, struct asy · line 248

Quoted 6 lines
kernel/async.c +6 -0
+async_cookie_t async_schedule_dev_domain(async_func_t func, struct device *dev,
+                                        struct async_domain *domain)
+{
+       return async_schedule_near_domain(func, dev, dev_to_node(dev), domain);
+}
+EXPORT_SYMBOL_GPL(async_schedule_dev_domain);

This seems unnecessary and restrictive. Callers may want to pass something other than dev as the parameter to the async function, and dev_to_node() is not on onerous burden to place on callers.

#14 lore raw

Re: [RFC workqueue/driver-core PATCH 2/5] async: Add support for queueing on specific NUMA node

Quoted 65 lines · Dan Williams
Quoted 20 lines · Alexander Duyck

This patch introduces four new variants of the async_schedule_ functions that allow scheduling on a specific NUMA node.

The first two functions are async_schedule_near and async_schedule_near_domain which end up mapping to async_schedule and async_schedule_domain but provide NUMA node specific functionality. They replace the original functions which were moved to inline function definitions that call the new functions while passing NUMA_NO_NODE.

The second two functions are async_schedule_dev and async_schedule_dev_domain which provide NUMA specific functionality when passing a device as the data member and that device has a NUMA node other than NUMA_NO_NODE.

The main motivation behind this is to address the need to be able to schedule device specific init work on specific NUMA nodes in order to improve performance of memory initialization.

[..]

Quoted 10 lines

/**
- * async_schedule - schedule a function for asynchronous execution
+ * async_schedule_near - schedule a function for asynchronous execution
* @func: function to execute asynchronously
* @data: data pointer to pass to the function
+ * @node: NUMA node that we want to schedule this on or close to *
* Returns an async_cookie_t that may be used for checkpointing later.
* Note: This function may be called from atomic or non-atomic contexts. */

kernel/async.c static async_cookie_t __async_schedule(async_func_t func, void *data, struct asy · line 232

Quoted 8 lines
kernel/async.c +3 -3
-async_cookie_t async_schedule(async_func_t func, void *data)
+async_cookie_t async_schedule_near(async_func_t func, void *data, int node)
  {
-       return __async_schedule(func, data, &async_dfl_domain);
+       return async_schedule_near_domain(func, data, node, &async_dfl_domain);
  }
-EXPORT_SYMBOL_GPL(async_schedule);
+EXPORT_SYMBOL_GPL(async_schedule_near);

Looks good to me. The _near() suffix makes it clear that we're doing a best effort hint to the work placement compared to the strict expectations of _on routines.

Quoted 19 lines

/**
- * async_schedule_domain - schedule a function for asynchronous execution within a certain domain
+ * async_schedule_dev_domain - schedule a function for asynchronous execution within a certain domain
* @func: function to execute asynchronously
- * @data: data pointer to pass to the function
+ * @dev: device that we are scheduling this work for
* @domain: the domain
*
- * Returns an async_cookie_t that may be used for checkpointing later.
- * @domain may be used in the async_synchronize_*_domain() functions to
- * wait within a certain synchronization domain rather than globally. A
- * synchronization domain is specified via @domain. Note: This function
- * may be called from atomic or non-atomic contexts.
+ * Device specific version of async_schedule_near_domain that provides some
+ * NUMA awareness based on the device node.
+ */

kernel/async.c +2 -0
+async_cookie_t async_schedule_dev_domain(async_func_t func, struct device *dev,
+                                        struct async_domain *domain)
Quoted 8 lines

kernel/async.c static async_cookie_t __async_schedule(async_func_t func, void *data, struct asy · line 248

Quoted 4 lines
kernel/async.c +4 -0
+{
+       return async_schedule_near_domain(func, dev, dev_to_node(dev), domain);
+}
+EXPORT_SYMBOL_GPL(async_schedule_dev_domain);

This seems unnecessary and restrictive. Callers may want to pass something other than dev as the parameter to the async function, and dev_to_node() is not on onerous burden to place on callers.

That is what async_schedule_near_domain is for, they can call that. The "dev" versions of the calls as just supposed to be helpers since one of the most common parameters to the async_schedule calls is a device, so I thought I would just put together a function that takes care of all this for us so I could drop an argument and avoid having to use dev_to_node everywhere.

#15 lore raw

Re: [RFC workqueue/driver-core PATCH 2/5] async: Add support for queueing on specific NUMA node

On Thu, Sep 27, 2018 at 8:24 AM Alexander Duyck
<[email protected]> wrote:
[..]

Quoted 20 lines
Quoted 10 lines
Quoted 10 lines

- * Returns an async_cookie_t that may be used for checkpointing later.
- * @domain may be used in the async_synchronize_*_domain() functions to
- * wait within a certain synchronization domain rather than globally. A
- * synchronization domain is specified via @domain. Note: This function
- * may be called from atomic or non-atomic contexts.
+ * Device specific version of async_schedule_near_domain that provides some
+ * NUMA awareness based on the device node.
+ */

kernel/async.c +2 -0
+async_cookie_t async_schedule_dev_domain(async_func_t func, struct device *dev,
+                                        struct async_domain *domain)
Quoted 8 lines

kernel/async.c static async_cookie_t __async_schedule(async_func_t func, void *data, struct asy · line 248

Quoted 4 lines
kernel/async.c +4 -0
+{
+       return async_schedule_near_domain(func, dev, dev_to_node(dev), domain);
+}
+EXPORT_SYMBOL_GPL(async_schedule_dev_domain);

This seems unnecessary and restrictive. Callers may want to pass something other than dev as the parameter to the async function, and dev_to_node() is not on onerous burden to place on callers.

Quoted 6 lines

That is what async_schedule_near_domain is for, they can call that. The "dev" versions of the calls as just supposed to be helpers since one of the most common parameters to the async_schedule calls is a device, so I thought I would just put together a function that takes care of all this for us so I could drop an argument and avoid having to use dev_to_node everywhere.

Yeah, makes sense, I guess I was reacting to the fact that this expands the number of exports unnecessarily. The other async routines are exported because they hide internal implementation details of the async implementation. The async_schedule_dev* helpers can just be static inline wrappers.

#16 lore raw

Re: [RFC workqueue/driver-core PATCH 2/5] async: Add support for queueing on specific NUMA node

Quoted 30 lines · Dan Williams

On Thu, Sep 27, 2018 at 8:24 AM Alexander Duyck
<[email protected]> wrote:
[..]

Quoted 20 lines
Quoted 10 lines
Quoted 10 lines

- * Returns an async_cookie_t that may be used for checkpointing later.
- * @domain may be used in the async_synchronize_*_domain() functions to
- * wait within a certain synchronization domain rather than globally. A
- * synchronization domain is specified via @domain. Note: This function
- * may be called from atomic or non-atomic contexts.
+ * Device specific version of async_schedule_near_domain that provides some
+ * NUMA awareness based on the device node.
+ */

kernel/async.c +2 -0
+async_cookie_t async_schedule_dev_domain(async_func_t func, struct device *dev,
+                                        struct async_domain *domain)
Quoted 8 lines

kernel/async.c static async_cookie_t __async_schedule(async_func_t func, void *data, struct asy · line 248

Quoted 4 lines
kernel/async.c +4 -0
+{
+       return async_schedule_near_domain(func, dev, dev_to_node(dev), domain);
+}
+EXPORT_SYMBOL_GPL(async_schedule_dev_domain);

This seems unnecessary and restrictive. Callers may want to pass something other than dev as the parameter to the async function, and dev_to_node() is not on onerous burden to place on callers.

Quoted 6 lines

That is what async_schedule_near_domain is for, they can call that. The "dev" versions of the calls as just supposed to be helpers since one of the most common parameters to the async_schedule calls is a device, so I thought I would just put together a function that takes care of all this for us so I could drop an argument and avoid having to use dev_to_node everywhere.

Quoted 5 lines

Yeah, makes sense, I guess I was reacting to the fact that this expands the number of exports unnecessarily. The other async routines are exported because they hide internal implementation details of the async implementation. The async_schedule_dev* helpers can just be static inline wrappers.

I can do them as inline wrappers for the next patch set. Shouldn't be too much of an issue.

Thanks.

- Alex

#17 lore raw

RFC PATCH 3/5 driver core: Probe devices asynchronously instead of the driver

In reply to Alexander Duyck

This change makes it so that we probe devices asynchronously instead of the driver. This results in us seeing the same behavior if the device is registered before the driver or after. This way we can avoid serializing the initialization should the driver not be loaded until after the devices have already been added.

The motivation behind this is that if we have a set of devices that take a significant amount of time to load we can greatly reduce the time to load by processing them in parallel instead of one at a time. In addition, each device can exist on a different node so placing a single thread on one CPU to initialize all of the devices for a given driver can result in poor performance on a system with multiple nodes.

One issue I can see with this patch is that I am using the dev_set/get_drvdata functions to store the driver in the device while I am waiting on the asynchronous init to complete. For now I am protecting it by using the lack of a dev->driver and the device lock.

---
 drivers/base/bus.c |   23 +++--------------------
 drivers/base/dd.c  |   42 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 45 insertions(+), 20 deletions(-)
drivers/base/bus.c +3 -20
index 8bfd27ec73d6..2a17bed657ec 100644
@@ -616,17 +616,6 @@ static ssize_t uevent_store(struct device_driver *drv, const char *buf,
 }
 static DRIVER_ATTR_WO(uevent);
 
-static void driver_attach_async(void *_drv, async_cookie_t cookie)
-{
-	struct device_driver *drv = _drv;
-	int ret;
-
-	ret = driver_attach(drv);
-
-	pr_debug("bus: '%s': driver %s async attach completed: %d\n",
-		 drv->bus->name, drv->name, ret);
-}
-
 /**
  * bus_add_driver - Add a driver to the bus.
  * @drv: driver.
@@ -659,15 +648,9 @@ int bus_add_driver(struct device_driver *drv)
 
 	klist_add_tail(&priv->knode_bus, &bus->p->klist_drivers);
 	if (drv->bus->p->drivers_autoprobe) {
-		if (driver_allows_async_probing(drv)) {
-			pr_debug("bus: '%s': probing driver %s asynchronously\n",
-				drv->bus->name, drv->name);
-			async_schedule(driver_attach_async, drv);
-		} else {
-			error = driver_attach(drv);
-			if (error)
-				goto out_unregister;
-		}
+		error = driver_attach(drv);
+		if (error)
+			goto out_unregister;
 	}
 	module_add_driver(drv->owner, drv);
 
drivers/base/dd.c +42 -0

54 lines not on this page · raw message

#18 lore raw

Re: [RFC workqueue/driver-core PATCH 3/5] driver core: Probe devices asynchronously instead of the driver

Quoted 20 lines · Alexander Duyck

This change makes it so that we probe devices asynchronously instead of the driver. This results in us seeing the same behavior if the device is registered before the driver or after. This way we can avoid serializing the initialization should the driver not be loaded until after the devices have already been added.

The motivation behind this is that if we have a set of devices that take a significant amount of time to load we can greatly reduce the time to load by processing them in parallel instead of one at a time. In addition, each device can exist on a different node so placing a single thread on one CPU to initialize all of the devices for a given driver can result in poor performance on a system with multiple nodes.

One issue I can see with this patch is that I am using the dev_set/get_drvdata functions to store the driver in the device while I am waiting on the asynchronous init to complete. For now I am protecting it by using the lack of a dev->driver and the device lock.

[..]

Quoted 8 lines
drivers/base/dd.c +4 -0
@@ -891,6 +914,25 @@ static int __driver_attach(struct device *dev, void *data)
                return ret;
        } /* ret > 0 means positive match */
 
+       if (driver_allows_async_probing(drv)) {
+               /*
+                * Instead of probing the device synchronously we will
+                * probe it asynchronously to allow for more parallelism.

drivers/base/dd.c static int __driver_attach(struct device *dev, void *data) · line 930

Quoted 10 lines
drivers/base/dd.c +10 -0
+                *
+                * We only take the device lock here in order to guarantee
+                * that the dev->driver and driver_data fields are protected
+                */
+               dev_dbg(dev, "scheduling asynchronous probe\n");
+               device_lock(dev);
+               if (!dev->driver) {
+                       get_device(dev);
+                       dev_set_drvdata(dev, drv);
+                       async_schedule(__driver_attach_async_helper, dev);

I'm not sure async drivers / sub-systems are ready for their devices to show up in parallel. While userspace should not be relying on kernel device names, people get upset when devices change kernel names from one boot to the next, and I can see this change leading to that scenario.

If a driver / sub-system wants more parallelism than what driver_allows_async_probing() provides it should do it locally, for example, like libata does.

#19 lore raw

Re: [RFC workqueue/driver-core PATCH 3/5] driver core: Probe devices asynchronously instead of the driver

Quoted 39 lines · Dan Williams
Quoted 20 lines · Alexander Duyck

This change makes it so that we probe devices asynchronously instead of the driver. This results in us seeing the same behavior if the device is registered before the driver or after. This way we can avoid serializing the initialization should the driver not be loaded until after the devices have already been added.

The motivation behind this is that if we have a set of devices that take a significant amount of time to load we can greatly reduce the time to load by processing them in parallel instead of one at a time. In addition, each device can exist on a different node so placing a single thread on one CPU to initialize all of the devices for a given driver can result in poor performance on a system with multiple nodes.

One issue I can see with this patch is that I am using the dev_set/get_drvdata functions to store the driver in the device while I am waiting on the asynchronous init to complete. For now I am protecting it by using the lack of a dev->driver and the device lock.

[..]

Quoted 16 lines
drivers/base/dd.c +12 -0
@@ -891,6 +914,25 @@ static int __driver_attach(struct device *dev, void *data)
                 return ret;
         } /* ret > 0 means positive match */
 
+       if (driver_allows_async_probing(drv)) {
+               /*
+                * Instead of probing the device synchronously we will
+                * probe it asynchronously to allow for more parallelism.
+                *
+                * We only take the device lock here in order to guarantee
+                * that the dev->driver and driver_data fields are protected
+                */
+               dev_dbg(dev, "scheduling asynchronous probe\n");
+               device_lock(dev);
+               if (!dev->driver) {
+                       get_device(dev);
Quoted 8 lines

drivers/base/dd.c static int __driver_attach(struct device *dev, void *data) · line 930

Quoted 2 lines
drivers/base/dd.c +2 -0
+                       dev_set_drvdata(dev, drv);
+                       async_schedule(__driver_attach_async_helper, dev);

I'm not sure async drivers / sub-systems are ready for their devices to show up in parallel. While userspace should not be relying on kernel device names, people get upset when devices change kernel names from one boot to the next, and I can see this change leading to that scenario.

The thing is the current async behavior already does this if the driver is loaded before the device is added. All I am doing is making the behavior with the driver loaded first the standard instead of letting it work the other way around. This way we get consistent behavior.

Quoted 3 lines

If a driver / sub-system wants more parallelism than what driver_allows_async_probing() provides it should do it locally, for example, like libata does.

So where I actually saw this was with the pmem legacy setup I had. After doing all the work to parallelize things in the driver it had no effect. That was because the nd_pmem driver wasn't loaded yet so all the device_add calls did is add the device but didn't attach the nd_pmem driver. Then when the driver loaded it serialized the probe calls resulting in it taking twice as long as it needed to in order to initialize the memory.

This seems to affect standard persistent memory as well. The only difference is that instead of probing the device on the first pass we kick it back and reprobe it in nd_pmem_probe/nd_pfn_probe in order to set the correct personality and that in turn allows us to asynchronously reschedule the work on the correct CPU and deserialize it.

#20 lore raw

Re: [RFC workqueue/driver-core PATCH 3/5] driver core: Probe devices asynchronously instead of the driver

Quoted 52 lines · Alexander Duyck
Quoted 39 lines · Dan Williams
Quoted 20 lines · Alexander Duyck

This change makes it so that we probe devices asynchronously instead of the driver. This results in us seeing the same behavior if the device is registered before the driver or after. This way we can avoid serializing the initialization should the driver not be loaded until after the devices have already been added.

The motivation behind this is that if we have a set of devices that take a significant amount of time to load we can greatly reduce the time to load by processing them in parallel instead of one at a time. In addition, each device can exist on a different node so placing a single thread on one CPU to initialize all of the devices for a given driver can result in poor performance on a system with multiple nodes.

One issue I can see with this patch is that I am using the dev_set/get_drvdata functions to store the driver in the device while I am waiting on the asynchronous init to complete. For now I am protecting it by using the lack of a dev->driver and the device lock.

[..]

Quoted 16 lines
drivers/base/dd.c +12 -0
@@ -891,6 +914,25 @@ static int __driver_attach(struct device *dev, void *data)
                 return ret;
         } /* ret > 0 means positive match */
 
+       if (driver_allows_async_probing(drv)) {
+               /*
+                * Instead of probing the device synchronously we will
+                * probe it asynchronously to allow for more parallelism.
+                *
+                * We only take the device lock here in order to guarantee
+                * that the dev->driver and driver_data fields are protected
+                */
+               dev_dbg(dev, "scheduling asynchronous probe\n");
+               device_lock(dev);
+               if (!dev->driver) {
+                       get_device(dev);
Quoted 8 lines

drivers/base/dd.c static int __driver_attach(struct device *dev, void *data) · line 930

Quoted 2 lines
drivers/base/dd.c +2 -0
+                       dev_set_drvdata(dev, drv);
+                       async_schedule(__driver_attach_async_helper, dev);

I'm not sure async drivers / sub-systems are ready for their devices to show up in parallel. While userspace should not be relying on kernel device names, people get upset when devices change kernel names from one boot to the next, and I can see this change leading to that scenario.

Quoted 4 lines

The thing is the current async behavior already does this if the driver is loaded before the device is added. All I am doing is making the behavior with the driver loaded first the standard instead of letting it work the other way around. This way we get consistent behavior.

Ok, I can see the consistency argument. It's still a behavior change that needs testing. Configurations that have been living with the default of synchronous probing of the devices on the bus for a later arriving driver might be surprised.

That said, I was confusing async probing with device registration in my thinking, so some of the discovery order / naming concerns may not be as bad as I was imagining. Sub-systems that would be broken by this behavior change would already be broken if a driver is built-in vs module.

So, consider this, an Acked-by.

Quoted 12 lines
Quoted 3 lines

If a driver / sub-system wants more parallelism than what driver_allows_async_probing() provides it should do it locally, for example, like libata does.

So where I actually saw this was with the pmem legacy setup I had. After doing all the work to parallelize things in the driver it had no effect. That was because the nd_pmem driver wasn't loaded yet so all the device_add calls did is add the device but didn't attach the nd_pmem driver. Then when the driver loaded it serialized the probe calls resulting in it taking twice as long as it needed to in order to initialize the memory.

Quoted 5 lines

This seems to affect standard persistent memory as well. The only difference is that instead of probing the device on the first pass we kick it back and reprobe it in nd_pmem_probe/nd_pfn_probe in order to set the correct personality and that in turn allows us to asynchronously reschedule the work on the correct CPU and deserialize it.

I don't see any problems with this series with the nvdimm unit tests, but note those tests do not check for discovery order / naming discrepancies.

#21 lore raw

RFC PATCH 4/5 driver core: Use new async_schedule_dev command

In reply to Alexander Duyck

This change makes it so that we use the device specific version of the async_schedule commands to defer various tasks related to devices. By doing this we should see a slight improvement in performance as any device that is sensitive to latency/locality in the setup will now be initializing on the node closest to the device.

---
 drivers/base/dd.c         |    4 ++--
 drivers/base/power/main.c |   12 ++++++------
 2 files changed, 8 insertions(+), 8 deletions(-)
drivers/base/dd.c +2 -2
index 5ba366c1cb83..81472dc44a70 100644
@@ -826,7 +826,7 @@ static int __device_attach(struct device *dev, bool allow_async)
 			 */
 			dev_dbg(dev, "scheduling asynchronous probe\n");
 			get_device(dev);
-			async_schedule(__device_attach_async_helper, dev);
+			async_schedule_dev(__device_attach_async_helper, dev);
 		} else {
 			pm_request_idle(dev);
 		}
@@ -927,7 +927,7 @@ static int __driver_attach(struct device *dev, void *data)
 		if (!dev->driver) {
 			get_device(dev);
 			dev_set_drvdata(dev, drv);
-			async_schedule(__driver_attach_async_helper, dev);
+			async_schedule_dev(__driver_attach_async_helper, dev);
 		}
 		device_unlock(dev);
 		return 0;
drivers/base/power/main.c +6 -6
index 3f68e2919dc5..8495d9b1e9d0 100644
@@ -726,7 +726,7 @@ void dpm_noirq_resume_devices(pm_message_t state)
 		reinit_completion(&dev->power.completion);
 		if (is_async(dev)) {
 			get_device(dev);
-			async_schedule(async_resume_noirq, dev);
+			async_schedule_dev(async_resume_noirq, dev);
 		}
 	}
 
@@ -883,7 +883,7 @@ void dpm_resume_early(pm_message_t state)
 		reinit_completion(&dev->power.completion);
 		if (is_async(dev)) {
 			get_device(dev);
-			async_schedule(async_resume_early, dev);
+			async_schedule_dev(async_resume_early, dev);
 		}
 	}
 
@@ -1047,7 +1047,7 @@ void dpm_resume(pm_message_t state)
 		reinit_completion(&dev->power.completion);
 		if (is_async(dev)) {
 			get_device(dev);
-			async_schedule(async_resume, dev);
+			async_schedule_dev(async_resume, dev);
 		}
 	}
 
@@ -1366,7 +1366,7 @@ static int device_suspend_noirq(struct device *dev)
 
 	if (is_async(dev)) {
 		get_device(dev);
-		async_schedule(async_suspend_noirq, dev);
+		async_schedule_dev(async_suspend_noirq, dev);
 		return 0;
 	}
 	return __device_suspend_noirq(dev, pm_transition, false);
@@ -1569,7 +1569,7 @@ static int device_suspend_late(struct device *dev)
 
 	if (is_async(dev)) {
 		get_device(dev);
-		async_schedule(async_suspend_late, dev);
+		async_schedule_dev(async_suspend_late, dev);
 		return 0;
 	}
 
@@ -1830,7 +1830,7 @@ static int device_suspend(struct device *dev)
 
 	if (is_async(dev)) {
 		get_device(dev);
-		async_schedule(async_suspend, dev);
+		async_schedule_dev(async_suspend, dev);
 		return 0;
 	}
#22 lore raw

Re: [RFC workqueue/driver-core PATCH 4/5] driver core: Use new async_schedule_dev command

Quoted 27 lines · Alexander Duyck

This change makes it so that we use the device specific version of the async_schedule commands to defer various tasks related to devices. By doing this we should see a slight improvement in performance as any device that is sensitive to latency/locality in the setup will now be initializing on the node closest to the device.

---
 drivers/base/dd.c         |    4 ++--
 drivers/base/power/main.c |   12 ++++++------
 2 files changed, 8 insertions(+), 8 deletions(-)
drivers/base/dd.c +1 -1
index 5ba366c1cb83..81472dc44a70 100644
@@ -826,7 +826,7 @@ static int __device_attach(struct device *dev, bool allow_async)
                         */
                        dev_dbg(dev, "scheduling asynchronous probe\n");
                        get_device(dev);
-                       async_schedule(__device_attach_async_helper, dev);
+                       async_schedule_dev(__device_attach_async_helper, dev);
                } else {
                        pm_request_idle(dev);
                }

drivers/base/dd.c static int __driver_attach(struct device *dev, void *data) · line 931

Quoted 6 lines
drivers/base/dd.c +1 -1
@@ -927,7 +927,7 @@ static int __driver_attach(struct device *dev, void *data)
                if (!dev->driver) {
                        get_device(dev);
                        dev_set_drvdata(dev, drv);
-                       async_schedule(__driver_attach_async_helper, dev);
+                       async_schedule_dev(__driver_attach_async_helper, dev);
                }

Above looks good to me...

drivers/base/power/main.c

Quoted 6 lines
drivers/base/dd.c +0 -0
                device_unlock(dev);
                return 0;
drivers/base/power/main.c +0 -0
index 3f68e2919dc5..8495d9b1e9d0 100644

Let's split these changes into their own patch, just in case it causes a suspend regression we can revert it independently of the async probing change. It might also be worthwhile to have a positive indication that this improves the latency of power transitions.

#23 lore raw

RFC PATCH 5/5 nvdimm: Schedule device registration on node local to the device

In reply to Alexander Duyck

This patch is meant to force the device registration for nvdimm devices to be closer to the actual device. This is achieved by using either the NUMA node ID of the region, or of the parent. By doing this we can have everything above the region based on the region, and everything below the region based on the nvdimm bus.

By guaranteeing NUMA locality I see an improvement of as high as 25% for per-node init of a system with 12TB of persistent memory.

---
 drivers/nvdimm/bus.c |   11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)
drivers/nvdimm/bus.c +8 -3
index 96154670bf07..f663d6ff524e 100644
@@ -23,6 +23,7 @@
 #include <linux/ndctl.h>
 #include <linux/sched.h>
 #include <linux/slab.h>
+#include <linux/cpu.h>
 #include <linux/fs.h>
 #include <linux/io.h>
 #include <linux/mm.h>
@@ -513,11 +514,15 @@ void __nd_device_register(struct device *dev)
 		set_dev_node(dev, to_nd_region(dev)->numa_node);
 
 	dev->bus = &nvdimm_bus_type;
-	if (dev->parent)
+	if (dev->parent) {
 		get_device(dev->parent);
+		if (dev_to_node(dev) == NUMA_NO_NODE)
+			set_dev_node(dev, dev_to_node(dev->parent));
+	}
 	get_device(dev);
-	async_schedule_domain(nd_async_device_register, dev,
-			&nd_async_domain);
+
+	async_schedule_dev_domain(nd_async_device_register, dev,
+				  &nd_async_domain);
 }
 
 void nd_device_register(struct device *dev)
#24 lore raw

Re: [RFC workqueue/driver-core PATCH 5/5] nvdimm: Schedule device registration on node local to the device

Quoted 11 lines · Alexander Duyck

This patch is meant to force the device registration for nvdimm devices to be closer to the actual device. This is achieved by using either the NUMA node ID of the region, or of the parent. By doing this we can have everything above the region based on the region, and everything below the region based on the nvdimm bus.

By guaranteeing NUMA locality I see an improvement of as high as 25% for per-node init of a system with 12TB of persistent memory.

Looks good to me. This does have a dependency on the current contents of libnvdimm development tree.