Skip to contentSkip to conversation

fastboot: keep at least one thread per cpu during boot

PATCH In review Frederic Weisbecker 3 participants 4 messages Feb 9, 2009

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 fastboot: keep at least one thread per cpu during boot

Async threads are created and destroyed depending on the number of jobs in queue. It means that several async threads can be created for a specific batch of work, then the threads will die after the completion of this batch, but they could be needed just after this completion for another batch of work.
During the boot, such repetitive thread creations can be wasteful, that's why this patch proposes to keep at least one thread per cpu (if they already have been created once). Such a threshold of threads kept alive will prevent from a part of the thread creation overhead.
This threshold will be dropped one the system_state switches from SYSTEM_BOOTING to SYSTEM_RUNNING.

Note:
_ If this patch is accepted, I will try to extend it to modules loading on boot _ One thread per cpu could sound a bit arbitrary here. Actually this is compromize between memory saving (if we just created lots of async thread for a large batch of jobs) and task creation overhead.

---
 include/linux/async.h |    2 +
 init/main.c           |    1 +
 kernel/async.c        |   52 +++++++++++++++++++++++++++++++++++++++++++++++-
 3 files changed, 53 insertions(+), 2 deletions(-)
include/linux/async.h +2 -0
index 68a9530..71a09e2 100644
@@ -25,3 +25,5 @@ extern void async_synchronize_cookie(async_cookie_t cookie);
 extern void async_synchronize_cookie_domain(async_cookie_t cookie,
 					    struct list_head *list);
 
+extern void async_finish_boot(void);
+
init/main.c +1 -0
index 36de89b..fa99928 100644
@@ -806,6 +806,7 @@ static noinline int init_post(void)
 	unlock_kernel();
 	mark_rodata_ro();
 	system_state = SYSTEM_RUNNING;
+	async_finish_boot();
 	numa_default_policy();
 
 	if (sys_open((const char __user *) "/dev/console", O_RDWR, 0) < 0)
kernel/async.c +50 -2

70 lines not on this page · raw message

#1 lore raw
Quoted 11 lines · On Mon, 9 Feb 2009 04:48:27 +0100 Frederic Weisbecker <[email protected]>…

Async threads are created and destroyed depending on the number of jobs in queue. It means that several async threads can be created for a specific batch of work, then the threads will die after the completion of this batch, but they could be needed just after this completion for another batch of work. During the boot, such repetitive thread creations can be wasteful, that's why this patch proposes to keep at least one thread per cpu (if they already have been created once). Such a threshold of threads kept alive will prevent from a part of the thread creation overhead. This threshold will be dropped one the system_state switches from SYSTEM_BOOTING to SYSTEM_RUNNING.

I'm not very fond of this to be honest;
at least during boot there's enough activity, and the time is so short (that's the point of the parallel stuff!) that this will not kick in to make a difference; specifically, every boot I've seen the number of threads is highest near the end, and also the total kernel boot time is below 1.5 seconds or so, not long enough for the threads to die.

Creating a thread is *CHEAP*. Really really cheap. You can do 100 thousands/second on even a modest CPU. If you have a high frequency of events, you don't want this, sure, and that is why there is a one second delay to give opportunity for reuse... but really....

Now, if async function calls get used more, I can see the point of always keeping one thread alive, just for both performance and VM low memory issues; but that's not what your patch is doing.

Signature · 3 lines
Arjan van de Ven 	Intel Open Source Technology Centre
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org
#2 lore raw
Quoted 22 lines · Arjan van de Ven
Quoted 11 lines · On Mon, 9 Feb 2009 04:48:27 +0100 Frederic Weisbecker <[email protected]>…

Async threads are created and destroyed depending on the number of jobs in queue. It means that several async threads can be created for a specific batch of work, then the threads will die after the completion of this batch, but they could be needed just after this completion for another batch of work. During the boot, such repetitive thread creations can be wasteful, that's why this patch proposes to keep at least one thread per cpu (if they already have been created once). Such a threshold of threads kept alive will prevent from a part of the thread creation overhead. This threshold will be dropped one the system_state switches from SYSTEM_BOOTING to SYSTEM_RUNNING.

I'm not very fond of this to be honest;
at least during boot there's enough activity, and the time is so short (that's the point of the parallel stuff!) that this will not kick in to make a difference; specifically, every boot I've seen the number of threads is highest near the end, and also the total kernel boot time is below 1.5 seconds or so, not long enough for the threads to die.

Quoted 4 lines

Creating a thread is *CHEAP*. Really really cheap. You can do 100 thousands/second on even a modest CPU. If you have a high frequency of events, you don't want this, sure, and that is why there is a one second delay to give opportunity for reuse... but really....

Agreed.

Quoted 5 lines

Now, if async function calls get used more, I can see the point of always keeping one thread alive, just for both performance and VM low memory issues; but that's not what your patch is doing.

I'd argue that the ability to _schedule_ async stuff without memory allocation would help more with low memory situations - after all, the work has been scheduled for later.

(For that, I have hacked up the following completely untested patch, but I'm not yet completely happy with it.)

---
 include/linux/async.h |   19 +++++++++
 kernel/async.c        |  105 +++++++++++++++++++++++++++++++++-----------------
 2 files changed, 89 insertions(+), 35 deletions(-)
include/linux/async.h +19 -0
@@ -16,9 +16,28 @@
 typedef u64 async_cookie_t;
 typedef void (async_func_ptr) (void *data, async_cookie_t cookie);
 
+/**
+ * struct async_entry - entry for asynchronous scheduling
+ * @list: anchor for internal lists
+ * @cookie: cookie for checkpointing
+ * @func: asynchronous function to execute
+ * @data: data to pass to the function
+ * @running: synchronization domain to use
+ * @persistent: 1 if the entry must not be deleted by the core
+ */
+struct async_entry {
+	struct list_head list;
+	async_cookie_t   cookie;
+	async_func_ptr	 *func;
+	void             *data;
+	struct list_head *running;
+	int              *persistent;
+};
+
 extern async_cookie_t async_schedule(async_func_ptr *ptr, void *data);
 extern async_cookie_t async_schedule_domain(async_func_ptr *ptr, void *data,
 					    struct list_head *list);
+extern async_cookie_t async_schedule_prealloc(struct async_entry *entry);
 extern void async_synchronize_full(void);
 extern void async_synchronize_full_domain(struct list_head *list);
 extern void async_synchronize_cookie(async_cookie_t cookie);
kernel/async.c +70 -35

149 lines not on this page · raw message

#3 lore raw
In reply to Arjan van de Ven
Quoted 15 lines · Arjan van de Ven
Quoted 11 lines · On Mon, 9 Feb 2009 04:48:27 +0100 Frederic Weisbecker <[email protected]>…

Async threads are created and destroyed depending on the number of jobs in queue. It means that several async threads can be created for a specific batch of work, then the threads will die after the completion of this batch, but they could be needed just after this completion for another batch of work. During the boot, such repetitive thread creations can be wasteful, that's why this patch proposes to keep at least one thread per cpu (if they already have been created once). Such a threshold of threads kept alive will prevent from a part of the thread creation overhead. This threshold will be dropped one the system_state switches from SYSTEM_BOOTING to SYSTEM_RUNNING.

Quoted 6 lines

I'm not very fond of this to be honest;
at least during boot there's enough activity, and the time is so short (that's the point of the parallel stuff!) that this will not kick in to make a difference; specifically, every boot I've seen the number of threads is highest near the end, and also the total kernel boot time is below 1.5 seconds or so, not long enough for the threads to die.

My boot takes more time (about 5 seconds before modules loading).

Quoted 4 lines

Creating a thread is *CHEAP*. Really really cheap. You can do 100 thousands/second on even a modest CPU. If you have a high frequency of events, you don't want this, sure, and that is why there is a one second delay to give opportunity for reuse... but really....

Ok. And that's a problem with my patch. I did not have a suitable testcase to produce a relevant benchmark: the aync insertions were to close in time to capture something interesting.
If I saw the result of a testcase, I would have seen probably no different :-)

I guess you're right, this would have added new code to maintain for only micro optimizations...

Quoted 4 lines

Now, if async function calls get used more, I can see the point of always keeping one thread alive, just for both performance and VM low memory issues; but that's not what your patch is doing.

Ok.
Perhaps the testcase would be suitable under embedeed systems. I will perhaps test one day :-)

Thanks!

Quoted 4 lines
Signature · 3 lines
Arjan van de Ven 	Intel Open Source Technology Centre
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org