perf: add workqueue library and use it in synthetic-events
cover perf: add workqueue library and use it in synthetic-events
This patchset introduces a new utility library inside perf/util, which provides a work queue abstraction, which loosely follows the Kernel workqueue API.
The workqueue abstraction is made up by two components:
- threadpool: which takes care of managing a pool of threads. It is inspired by the prototype for threaded trace in perf-record from Alexey:
https://lore.kernel.org/lkml/[email protected]/
- workqueue: manages a shared queue and provides the workers implementation.
On top of the workqueue, a simple parallel-for utility is implemented which is then showcased in synthetic-events.c, replacing the previous manual pthread-created threads.
Through some experiments with perf bench, I can see how the new workqueue has a higher overhead compared to manual creation of threads, but is able to more effectively partition work among threads, yielding a better result with more threads.
Furthermore, the overhead could be configured by changing the work_size (currently 1), aka the number of dirents that are processed by a thread before grabbing a lock to get the new work item. I experimented with different sizes but, while bigger sizes reduce overhead as expected, they do not scale as well to more threads.
I tried to keep the patchset as simple as possible, deferring possible improvements and features to future work.
Naming a few:
- in order to achieve a better performance, we could consider using work-stealing instead of a common queue.
- affinities in the thread pool, as in Alexey prototype for perf-record. Doing so would enable reusing the same threadpool for different purposes (evlist open, threaded trace, synthetic threads), avoiding having to spin up threads multiple times.
- resizable threadpool, e.g. for lazy spawining of threads.
@Arnaldo
Since I wanted the workqueue to provide a similar API to the Kernel's workqueue, I followed the naming style I found there, instead of the usual object__method style that is typically found in perf. Let me know if you'd like me to follow perf style instead.
Thanks,
Riccardo
Riccardo Mancini (10):
perf workqueue: threadpool creation and destruction perf tests: add test for workqueue
perf workqueue: add threadpool start and stop functions perf workqueue: add threadpool execute and wait functions perf workqueue: add sparse annotation header
perf workqueue: introduce workqueue struct
perf workqueue: implement worker thread and management perf workqueue: add queue_work and flush_workqueue functions perf workqueue: add utility to execute a for loop in parallel perf synthetic-events: use workqueue parallel_for
tools/perf/tests/Build | 1 + tools/perf/tests/builtin-test.c | 9 + tools/perf/tests/tests.h | 3 + tools/perf/tests/workqueue.c | 453 +++++++++++++++++ tools/perf/util/Build | 1 + tools/perf/util/synthetic-events.c | 131 +++-- tools/perf/util/workqueue/Build | 2 + tools/perf/util/workqueue/sparse.h | 21 + tools/perf/util/workqueue/threadpool.c | 516 ++++++++++++++++++++ tools/perf/util/workqueue/threadpool.h | 29 ++ tools/perf/util/workqueue/workqueue.c | 642 +++++++++++++++++++++++++ tools/perf/util/workqueue/workqueue.h | 38 ++ 12 files changed, 1771 insertions(+), 75 deletions(-) create mode 100644 tools/perf/tests/workqueue.c create mode 100644 tools/perf/util/workqueue/Build create mode 100644 tools/perf/util/workqueue/sparse.h create mode 100644 tools/perf/util/workqueue/threadpool.c create mode 100644 tools/perf/util/workqueue/threadpool.h create mode 100644 tools/perf/util/workqueue/workqueue.c create mode 100644 tools/perf/util/workqueue/workqueue.h
Signature · 1 line
2.31.1
RFC PATCH 1/10 perf workqueue: threadpool creation and destruction
The workqueue library is made up by two components:
- threadpool: handles the lifetime of the threads
- workqueue: handles work distribution among the threads
This first patch introduces the threadpool, starting from its creation and destruction functions.
Thread management is based on the prototype from Alexey:
https://lore.kernel.org/lkml/[email protected]/
Each thread in the threadpool executes the same function (aka task) with a different argument tidx.
Threads use a pair of pipes to communicate with the main process. The threadpool is static (all threads will be spawned at the same time). Future work could include making it resizable and adding affinity support (as in Alexey prototype).
--- tools/perf/util/Build | 1 + tools/perf/util/workqueue/Build | 1 + tools/perf/util/workqueue/threadpool.c | 175 +++++++++++++++++++++++++ tools/perf/util/workqueue/threadpool.h | 19 +++ 4 files changed, 196 insertions(+) create mode 100644 tools/perf/util/workqueue/Build create mode 100644 tools/perf/util/workqueue/threadpool.c create mode 100644 tools/perf/util/workqueue/threadpool.h
tools/perf/util/Build +1 -0
@@ -180,6 +180,7 @@ perf-$(CONFIG_LIBBABELTRACE) += data-convert-bt.o perf-y += data-convert-json.o perf-y += scripting-engines/ +perf-y += workqueue/ perf-$(CONFIG_ZLIB) += zlib.o perf-$(CONFIG_LZMA) += lzma.o
tools/perf/util/workqueue/Build new file +1 -0
@@ -0,0 +1 @@ +perf-y += threadpool.o
tools/perf/util/workqueue/threadpool.c new file +175 -0
175 lines not on this page · raw message
tools/perf/util/workqueue/threadpool.h new file +19 -0
@@ -0,0 +1,19 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __WORKQUEUE_THREADPOOL_H +#define __WORKQUEUE_THREADPOOL_H + +struct threadpool_struct ; +struct task_struct ; + +typedef void (*task_func_t )(int tidx,struct task_struct *task); + +struct task_struct { +task_func_t fn; +}; + +extern struct threadpool_struct *create_threadpool (int n_threads); +extern void destroy_threadpool (struct threadpool_struct *pool); + +extern int threadpool_size (struct threadpool_struct *pool); + +#endif /* __WORKQUEUE_THREADPOOL_H */
Signature · 1 line
2.31.1
RFC PATCH 3/10 perf workqueue: add threadpool start and stop functions
In reply to Riccardo ManciniThis patch adds the start and stop functions, alongside the thread function.
Each thread will run until a stop signal is received. Furthermore, start and stop are added to the test.
Thread management is based on the prototype from Alexey:
https://lore.kernel.org/lkml/[email protected]/
--- tools/perf/tests/workqueue.c | 13 ++ tools/perf/util/workqueue/threadpool.c | 238 +++++++++++++++++++++++++ tools/perf/util/workqueue/threadpool.h | 5 + 3 files changed, 256 insertions(+)
tools/perf/tests/workqueue.c +13 -0
@@ -10,16 +10,29 @@ struct threadpool_test_args_t {static int __threadpool__prepare (struct threadpool_struct **pool,int pool_size) { +int ret; + *pool =create_threadpool (pool_size);TEST_ASSERT_VAL ("threadpool creation failure" , *pool !=NULL );TEST_ASSERT_VAL ("threadpool size is wrong" ,threadpool_size (*pool) == pool_size); + ret =start_threadpool (*pool); +TEST_ASSERT_VAL ("threadpool start failure" , ret ==0 ); +TEST_ASSERT_VAL ("threadpool is not ready" ,threadpool_is_ready (*pool)); +return 0 ; }static int __threadpool__teardown (struct threadpool_struct *pool) { +int ret; + + ret =stop_threadpool (pool); +TEST_ASSERT_VAL ("threadpool start failure" , ret ==0 ); +TEST_ASSERT_VAL ("stopped threadpool is ready" , + !threadpool_is_ready (pool)); +destroy_threadpool (pool);return 0 ;
tools/perf/util/workqueue/threadpool.c +238 -0
265 lines not on this page · raw message
tools/perf/util/workqueue/threadpool.h +5 -0
@@ -14,6 +14,11 @@ struct task_struct {extern struct threadpool_struct *create_threadpool (int n_threads);extern void destroy_threadpool (struct threadpool_struct *pool); +extern int start_threadpool (struct threadpool_struct *pool); +extern int stop_threadpool (struct threadpool_struct *pool); +extern int threadpool_size (struct threadpool_struct *pool); +extern bool threadpool_is_ready (struct threadpool_struct *pool); +#endif /* __WORKQUEUE_THREADPOOL_H */
Signature · 1 line
2.31.1
RFC PATCH 2/10 perf tests: add test for workqueue
In reply to Riccardo ManciniIt will have subtests testing threadpool and workqueue separately. This patch only introduces the first subtest, checking that the threadpool is correctly created and destructed.
This test will be expanded when new functions are added in next patches.
--- tools/perf/tests/Build | 1 + tools/perf/tests/builtin-test.c | 9 +++ tools/perf/tests/tests.h | 3 + tools/perf/tests/workqueue.c | 113 ++++++++++++++++++++++++++++++++ 4 files changed, 126 insertions(+) create mode 100644 tools/perf/tests/workqueue.c
tools/perf/tests/Build +1 -0
@@ -64,6 +64,7 @@ perf-y += parse-metric.o perf-y += pe-file-parsing.o perf-y += expand-cgroup.o perf-y += perf-time-to-tsc.o +perf-y += workqueue.o $(OUTPUT)tests/llvm-src-base.c: tests/bpf-script-example.c tests/Build $(call rule_mkdir)
tools/perf/tests/builtin-test.c +9 -0
@@ -360,6 +360,15 @@ static struct test generic_tests[] = { .func = test__perf_time_to_tsc, .is_supported = test__tsc_is_supported, }, + { + .desc = "Test workqueue lib", + .func = test__workqueue, + .subtest = { + .skip_if_fail = false, + .get_nr = test__workqueue_subtest_get_nr, + .get_desc = test__workqueue_subtest_get_desc, + } + }, { .func = NULL, },
tools/perf/tests/tests.h +3 -0
@@ -127,6 +127,9 @@ int test__parse_metric(struct test *test, int subtest);int test__pe_file_parsing (struct test *test,int subtest);int test__expand_cgroup_events (struct test *test,int subtest);int test__perf_time_to_tsc (struct test *test,int subtest); +int test__workqueue (struct test *test,int subtest); +const char *test__workqueue_subtest_get_desc (int subtest); +int test__workqueue_subtest_get_nr (void );bool test__bp_signal_is_supported (void );bool test__bp_account_is_supported (void );
tools/perf/tests/workqueue.c new file +113 -0
113 lines not on this page · raw message
Signature · 1 line
2.31.1
RFC PATCH 4/10 perf workqueue: add threadpool execute and wait functions
In reply to Riccardo ManciniThis patch adds:
- execute_in_threadpool: assigns a task to the threads to execute asynchronously.
- wait_threadpool: waits for the task to complete on all threads.
Furthermore, testing for these new functions is added.
This patch completes the threadpool.
--- tools/perf/tests/workqueue.c | 86 ++++++++++++++++++++- tools/perf/util/workqueue/threadpool.c | 103 +++++++++++++++++++++++++ tools/perf/util/workqueue/threadpool.h | 5 ++ 3 files changed, 193 insertions(+), 1 deletion(-)
tools/perf/tests/workqueue.c +85 -1
@@ -1,13 +1,59 @@// SPDX-License-Identifier: GPL-2.0 +#include <stdlib.h> #include <linux/kernel.h> +#include <linux/zalloc.h> #include "tests.h" #include "util/debug.h" #include "util/workqueue/threadpool.h" +#define DUMMY_FACTOR 100000 +#define N_DUMMY_WORK_SIZES 7 +struct threadpool_test_args_t {int pool_size; }; +struct test_task { +struct task_struct task; +int n_threads; +int *array; +}; + +/** +* dummy_work - calculates DUMMY_FACTOR * (idx % N_DUMMY_WORK_SIZES) inefficiently +* +* This function uses modulus to create work items of different sizes. +*/ +static void dummy_work (int idx) +{ +int prod =0 ; +int k = idx %N_DUMMY_WORK_SIZES ; +int i, j; + +for (i =0 ; i <DUMMY_FACTOR ; i++) +for (j =0 ; j < k; j++) + prod ++; + +pr_debug3 ("dummy: %d * %d = %d\n" ,DUMMY_FACTOR , k, prod); +} + +static void test_task_fn1 (int tidx,struct task_struct *task) +{ + struct test_task *mtask = container_of(task, struct test_task, task); + +dummy_work (tidx); + mtask->array[tidx] = tidx+1 ; +} + +static void test_task_fn2 (int tidx,struct task_struct *task) +{ + struct test_task *mtask = container_of(task, struct test_task, task); + +dummy_work (tidx); + mtask->array[tidx] = tidx*2 ; +} + +static int __threadpool__prepare (struct threadpool_struct **pool,int pool_size) {int ret;
@@ -38,21 +84,59 @@ static int __threadpool__teardown(struct threadpool_struct *pool)return 0 ; } +static int __threadpool__exec_wait (struct threadpool_struct *pool, +struct task_struct *task) +{ +int ret; + + ret =execute_in_threadpool (pool, task); +TEST_ASSERT_VAL ("threadpool execute failure" , ret ==0 ); +TEST_ASSERT_VAL ("threadpool is not executing" ,threadpool_is_busy (pool)); + + ret =wait_threadpool (pool); +TEST_ASSERT_VAL ("threadpool wait failure" , ret ==0 ); +TEST_ASSERT_VAL ("waited threadpool is not ready" ,threadpool_is_ready (pool)); + +return 0 ; +}static int __test__threadpool (void *_args) {struct threadpool_test_args_t *args = _args;struct threadpool_struct *pool; -int ret; +int ret, i; +struct test_task task; + + task.task.fn = test_task_fn1; + task.n_threads = args->pool_size; + task.array =calloc (args->pool_size,sizeof (*task.array)); ret =__threadpool__prepare (&pool, args->pool_size);if (ret)return ret; + ret =__threadpool__exec_wait (pool, &task.task); +if (ret) +return ret; + +for (i =0 ; i < args->pool_size; i++) +TEST_ASSERT_VAL ("failed array check (1)" , task.array[i] == i+1 ); + + task.task.fn = test_task_fn2; + + ret =__threadpool__exec_wait (pool, &task.task); +if (ret) +return ret; + +for (i =0 ; i < args->pool_size; i++) +TEST_ASSERT_VAL ("failed array check (2)" , task.array[i] ==2 *i); + ret =__threadpool__teardown (pool);if (ret)return ret; +free (task.array); +return 0 ; }
tools/perf/util/workqueue/threadpool.c +103 -0
135 lines not on this page · raw message
tools/perf/util/workqueue/threadpool.h +5 -0
@@ -17,8 +17,13 @@ extern void destroy_threadpool(struct threadpool_struct *pool);extern int start_threadpool (struct threadpool_struct *pool);extern int stop_threadpool (struct threadpool_struct *pool); +extern int execute_in_threadpool (struct threadpool_struct *pool, +struct task_struct *task); +extern int wait_threadpool (struct threadpool_struct *pool); +extern int threadpool_size (struct threadpool_struct *pool);extern bool threadpool_is_ready (struct threadpool_struct *pool); +extern int threadpool_is_busy (struct threadpool_struct *pool);#endif /* __WORKQUEUE_THREADPOOL_H */
Signature · 1 line
2.31.1
RFC PATCH 5/10 perf workqueue: add sparse annotation header
In reply to Riccardo ManciniThis patch adds a simple header containing sparse annotations.
TODO: what is the best place to put this?
--- tools/perf/util/workqueue/sparse.h | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100644 tools/perf/util/workqueue/sparse.h
tools/perf/util/workqueue/sparse.h new file +21 -0
@@ -0,0 +1,21 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __WORKQUEUE_SPARSE_H +#define __WORKQUEUE_SPARSE_H + +#ifdef __CHECKER__ +# define __must_hold (x) __attribute__((context(x, 1, 1))) +# define __acquires (x) __attribute__((context(x, 0, 1))) +# define __releases (x) __attribute__((context(x, 1, 0))) +# define __acquire (x) __context__(x, 1) +# define __release (x) __context__(x, -1) +# define __cond_lock (x, c) ((c) ? ({ __acquire(x); 1; }) : 0) +#else +# define __must_hold (x) +# define __acquires (x) +# define __releases (x) +# define __acquire (x) ((void)0) +# define __release (x) ((void)0) +# define __cond_lock (x, c) (c) +#endif + +#endif /* __WORKQUEUE_SPARSE_H */
Signature · 1 line
2.31.1
RFC PATCH 7/10 perf workqueue: implement worker thread and management
In reply to Riccardo ManciniThis patch adds the implementation of the worker thread that is executed in the threadpool, and all management-related functions.
At startup, a worker registers itself with the workqueue by adding itself to the idle_list, then it sends an ack back to the main thread. When creating wotkers, the main thread will wait for the related acks. Once there is work to do, threads are woken up to perform the work. Threads will try to dequeue a new pending work before going to sleep.
This registering mechanism has been implemented to enable for dynamic growth of the workqueue in the future.
--- tools/perf/util/workqueue/workqueue.c | 244 +++++++++++++++++++++++++- 1 file changed, 242 insertions(+), 2 deletions(-)
tools/perf/util/workqueue/workqueue.c +242 -2
@@ -7,8 +7,18 @@#include <pthread.h> #include <linux/list.h> #include "debug.h" +#include "sparse.h" #include "workqueue.h" +enum worker_msg { +WORKER_MSG__UNDEFINED , +WORKER_MSG__READY ,/* from worker: ack */ +WORKER_MSG__WAKE ,/* to worker: wake up */ +WORKER_MSG__STOP ,/* to worker: exit */ +WORKER_MSG__ERROR , +WORKER_MSG__MAX +}; +enum workqueue_status {WORKQUEUE_STATUS__READY ,/* wq is ready to receive work */ WORKQUEUE_STATUS__ERROR ,
@@ -27,14 +37,217 @@ struct workqueue_struct {enum workqueue_status status; }; +struct worker { +int tidx;/* idx of thread in pool */ +struct list_head entry;/* in idle or busy list */ +struct work_struct *current_work;/* work being processed */ +int msg_pipe[2 ];/* main thread comm pipes*/ +}; + +#define for_each_busy_worker (wq, m_worker) \ + list_for_each_entry(m_worker, &wq->busy_list, entry) + +#define for_each_idle_worker (wq, m_worker) \ + list_for_each_entry(m_worker, &wq->idle_list, entry) + +static inline int lock_workqueue (struct workqueue_struct *wq) +__acquires(&wq->lock) +{ +__acquire (&wq->lock); +return pthread_mutex_lock (&wq->lock); +} + +static inline int unlock_workqueue (struct workqueue_struct *wq) +__releases(&wq->lock) +{ +__release (&wq->lock); +return pthread_mutex_unlock (&wq->lock); +} + +/** +* available_work - check if @wq has work to do +*/ +static int available_work (struct workqueue_struct *wq) +__must_hold(&wq->lock) +{ +return !list_empty (&wq->pending); +} + +/** +* dequeue_work - retrieve the next work in @wq to be executed by the worker +* +* Called inside worker. +*/ +static struct work_struct *dequeue_work (struct workqueue_struct *wq) +__must_hold(&wq->lock) +{ + struct work_struct *work = list_first_entry(&wq->pending, struct work_struct, entry); + +list_del_init (&work->entry); +return work; +} + +/** +* sleep_worker - worker @w of workqueue @wq goes to sleep +* +* Called inside worker. +* If this was the last idle thread, signal it to the main thread, in case it +* was flushing the workqueue. +*/ +static void sleep_worker (struct workqueue_struct *wq,struct worker *w) +__must_hold(&wq->lock) +{ +list_move (&w->entry, &wq->idle_list); +if (list_empty (&wq->busy_list)) +pthread_cond_signal (&wq->idle_cond); +} + +/** +* stop_worker - stop worker @w +* +* Called from main thread. +* Send stop message to worker @w. +*/ +static int stop_worker (struct worker *w) +{ +int ret; +enum worker_msg msg; + + msg =WORKER_MSG__STOP ; + ret =write (w->msg_pipe[1 ], &msg,sizeof (msg)); +if (ret <0 ) { +pr_err ("workqueue: error sending stop msg: %s\n" , +strerror (errno)); +return -1 ; + } + +return 0 ; +} + +/** +* init_worker - init @w struct +* @w: the struct to init +* @tidx: index of the executing thread inside the threadpool +*/ +static int init_worker (struct worker *w,int tidx) +{ +if (pipe (w->msg_pipe)) { +pr_err ("worker[%d]: error opening pipe: %s\n" , tidx,strerror (errno)); +return -1 ; + } + + w->tidx = tidx; + w->current_work =NULL ; +INIT_LIST_HEAD (&w->entry); + +return 0 ; +} + +/** +* fini_worker - deallocate resources used by @w struct +*/ +static void fini_worker (struct worker *w) +{ +close (w->msg_pipe[0 ]); + w->msg_pipe[0 ] =-1 ; +close (w->msg_pipe[1 ]); + w->msg_pipe[1 ] =-1 ; +} + +/** +* register_worker - add worker to @wq->idle_list +*/ +static void register_worker (struct workqueue_struct *wq,struct worker *w) +__must_hold(&wq->lock) +{ +list_move (&w->entry, &wq->idle_list); +} + +/** +* unregister_worker - remove worker from @wq->idle_list +*/ +static void unregister_worker (struct workqueue_struct *wq__maybe_unused , +struct worker *w) +__must_hold(&wq->lock) +{ +list_del_init (&w->entry); +} +/** * worker_thread - worker function executed on threadpool */ static void worker_thread (int tidx,struct task_struct *task) { struct workqueue_struct *wq = container_of(task, struct workqueue_struct, task); +struct worker this_worker; +enum worker_msg msg; +int ret, init_err; + + init_err =init_worker (&this_worker, tidx); +if (init_err) { +// send error message to main thread + msg =WORKER_MSG__ERROR ; + }else { +lock_workqueue (wq); +register_worker (wq, &this_worker); +unlock_workqueue (wq); + +// ack worker creation + msg =WORKER_MSG__READY ; + } + + ret =write (wq->msg_pipe[1 ], &msg,sizeof (msg)); +if (ret <0 ) { +pr_err ("worker[%d]: error sending msg: %s\n" , + tidx,strerror (errno)); + +if (init_err) +return ; + goto out; + } -pr_debug ("hi from worker %d. Pool is in status %d\n" , tidx, wq->status); +// stop if there have been errors in init +if (init_err) +return ; + +for (;;) { + msg =WORKER_MSG__UNDEFINED ; + ret =read (this_worker.msg_pipe[0 ], &msg,sizeof (msg)); +if (ret <0 || (msg !=WORKER_MSG__WAKE && msg !=WORKER_MSG__STOP )) { +pr_err ("worker[%d]: error receiving msg: %s\n" , + tidx,strerror (errno)); +break ; + } + +if (msg ==WORKER_MSG__STOP ) +break ; + +// main thread takes care of moving to busy list and assigning current_work + +while (this_worker.current_work) { + this_worker.current_work->func (this_worker.current_work); + +lock_workqueue (wq); +if (available_work (wq)) { + this_worker.current_work =dequeue_work (wq); +pr_debug ("worker[%d]: dequeued work\n" , + tidx); + }else { + this_worker.current_work =NULL ; +sleep_worker (wq, &this_worker); +pr_debug ("worker[%d]: going to sleep\n" , + tidx); + } +unlock_workqueue (wq); + } + } + +out: +lock_workqueue (wq); +unregister_worker (wq, &this_worker); +unlock_workqueue (wq); + +fini_worker (&this_worker); }/**
@@ -43,7 +256,8 @@ static void worker_thread(int tidx, struct task_struct *task)static int attach_threadpool_to_workqueue (struct workqueue_struct *wq,struct threadpool_struct *pool) { -int err; +int err, ret, t; +enum worker_msg msg;if (!threadpool_is_ready (pool)) {pr_err ("workqueue: cannot attach to pool: pool is not ready\n" );
@@ -56,6 +270,22 @@ static int attach_threadpool_to_workqueue(struct workqueue_struct *wq,if (err)return -1 ; + +// wait ack from all threads +for (t =0 ; t <threadpool_size (pool); t++) { + msg =WORKER_MSG__UNDEFINED ; + ret =read (wq->msg_pipe[0 ], &msg,sizeof (msg)); +if (ret <0 ) { +pr_err ("workqueue: error receiving ack: %s\n" , +strerror (errno)); +return -1 ; + } +if (msg !=WORKER_MSG__READY ) { +pr_err ("workqueue: received error\n" ); +return -1 ; + } + } +return 0 ; }
@@ -65,12 +295,22 @@ static int attach_threadpool_to_workqueue(struct workqueue_struct *wq,static int detach_threadpool_from_workqueue (struct workqueue_struct *wq) {int ret, err =0 ; +struct worker *w;if (wq->status !=WORKQUEUE_STATUS__READY ) {pr_err ("workqueue: cannot attach to pool: wq is not ready\n" );return -1 ; } +lock_workqueue (wq); +for_each_idle_worker (wq, w) { + ret =stop_worker (w); +if (ret) + err =-1 ; + } +unlock_workqueue (wq); + + ret =wait_threadpool (wq->pool);if (ret) {pr_err ("workqueue: error waiting threadpool\n" );
Signature · 1 line
2.31.1
RFC PATCH 8/10 perf workqueue: add queue_work and flush_workqueue functions
In reply to Riccardo ManciniThis patch adds functions to queue and wait work_structs, and related tests.
When a new work item is added, the workqueue first checks if there are threads to wake up. If so, it wakes it up with the given work item, otherwise it will just add it to the list pending work items. A thread which completes a work item will check this list before going to sleep.
--- tools/perf/tests/workqueue.c | 69 +++++++++++++++++++- tools/perf/util/workqueue/workqueue.c | 93 +++++++++++++++++++++++++++ tools/perf/util/workqueue/workqueue.h | 7 ++ 3 files changed, 168 insertions(+), 1 deletion(-)
tools/perf/tests/workqueue.c +68 -1
@@ -146,6 +146,27 @@ static int __test__threadpool(void *_args)return 0 ; } +struct test_work { +struct work_struct work; +int i; +int *array; +}; + +static void test_work_fn1 (struct work_struct *work) +{ + struct test_work *mwork = container_of(work, struct test_work, work); + +dummy_work (mwork->i); + mwork->array[mwork->i] = mwork->i+1 ; +} + +static void test_work_fn2 (struct work_struct *work) +{ + struct test_work *mwork = container_of(work, struct test_work, work); + +dummy_work (mwork->i); + mwork->array[mwork->i] = mwork->i*2 ; +} static int __workqueue__prepare(struct threadpool_struct **pool, struct workqueue_struct **wq,
@@ -180,21 +201,67 @@ static int __workqueue__teardown(struct threadpool_struct *pool,return 0 ; } +static int __workqueue__exec_wait (struct workqueue_struct *wq, +int *array,struct test_work *works, +work_func_t func,int n_work_items) +{ +int ret, i; + +for (i =0 ; i < n_work_items; i++) { + works[i].array = array; + works[i].i = i; + +init_work (&works[i].work); + works[i].work.func = func; +queue_work (wq, &works[i].work); + } + + ret =flush_workqueue (wq); +TEST_ASSERT_VAL ("workqueue flush failure" , ret ==0 ); + +return 0 ; +} + +static int __test__workqueue (void *_args) {struct workqueue_test_args_t *args = _args;struct threadpool_struct *pool;struct workqueue_struct *wq; -int ret; +int ret, i; +int *array; +struct test_work *works; + + array =calloc (args->n_work_items,sizeof (*array)); + works =calloc (args->n_work_items,sizeof (*works)); ret =__workqueue__prepare (&pool, &wq, args->pool_size);if (ret)return ret; + ret =__workqueue__exec_wait (wq, array, works, test_work_fn1, + args->n_work_items); +if (ret) +return ret; + +for (i =0 ; i < args->n_work_items; i++) +TEST_ASSERT_VAL ("failed array check (1)" , array[i] == i+1 ); + + ret =__workqueue__exec_wait (wq, array, works, test_work_fn2, + args->n_work_items); +if (ret) +return ret; + +for (i =0 ; i < args->n_work_items; i++) +TEST_ASSERT_VAL ("failed array check (2)" , array[i] ==2 *i); + ret =__workqueue__teardown (pool, wq);if (ret)return ret; +free (array); +free (works); +return 0 ; }
tools/perf/util/workqueue/workqueue.c +93 -0
114 lines not on this page · raw message
tools/perf/util/workqueue/workqueue.h +7 -0
@@ -21,4 +21,11 @@ extern struct workqueue_struct *create_workqueue(struct threadpool_struct *pool)extern int destroy_workqueue (struct workqueue_struct *wq);extern int workqueue_nr_threads (struct workqueue_struct *wq); + +extern int queue_work (struct workqueue_struct *wq,struct work_struct *work); + +extern int flush_workqueue (struct workqueue_struct *wq); + +extern void init_work (struct work_struct *work); +#endif /* __WORKQUEUE_WORKQUEUE_H */
Signature · 1 line
2.31.1
RFC PATCH 9/10 perf workqueue: add utility to execute a for loop in parallel
In reply to Riccardo ManciniThis patch adds the parallel_for which executes a given function inside the workqueue, taking care of managing the work items.
--- tools/perf/tests/workqueue.c | 84 +++++++++++++++++ tools/perf/util/workqueue/workqueue.c | 125 ++++++++++++++++++++++++++ tools/perf/util/workqueue/workqueue.h | 7 ++ 3 files changed, 216 insertions(+)
tools/perf/tests/workqueue.c +84 -0
@@ -19,6 +19,12 @@ struct workqueue_test_args_t {int n_work_items; }; +struct parallel_for_test_args_t { +int pool_size; +int n_work_items; +int work_size; +}; +struct test_task {struct task_struct task;int n_threads;
@@ -265,6 +271,44 @@ static int __test__workqueue(void *_args)return 0 ; } +static void test_pfw_fn (int i,void *args) +{ +int *array = args; + +dummy_work (i); + array[i] = i+1 ; +} + +static int __test__parallel_for (void *_args) +{ +struct parallel_for_test_args_t *args = _args; +struct threadpool_struct *pool; +struct workqueue_struct *wq; +int ret, i; +int *array; + + array =calloc (args->n_work_items,sizeof (*array)); + + ret =__workqueue__prepare (&pool, &wq, args->pool_size); +if (ret) +return ret; + + ret =parallel_for (wq,0 , args->n_work_items, args->work_size, + test_pfw_fn, array); +TEST_ASSERT_VAL ("parallel_for failure" , ret ==0 ); + +for (i =0 ; i < args->n_work_items; i++) +TEST_ASSERT_VAL ("failed array check" , array[i] == i+1 ); + + ret =__workqueue__teardown (pool, wq); +if (ret) +return ret; + +free (array); + +return 0 ; +} +static const struct threadpool_test_args_t threadpool_test_args[] = { { .pool_size =1
@@ -310,6 +354,39 @@ static const struct workqueue_test_args_t workqueue_test_args[] = { } }; +static const struct parallel_for_test_args_t parallel_for_test_args[] = { + { + .pool_size =1 , + .n_work_items =1 , + .work_size =1 + }, + { + .pool_size =1 , + .n_work_items =10 , + .work_size =3 + }, + { + .pool_size =2 , + .n_work_items =1 , + .work_size =1 + }, + { + .pool_size =2 , + .n_work_items =100 , + .work_size =10 + }, + { + .pool_size =16 , + .n_work_items =7 , + .work_size =2 + }, + { + .pool_size =16 , + .n_work_items =2789 , + .work_size =16 + } +}; +struct test_case {const char *desc;int (*func)(void *args);
@@ -332,6 +409,13 @@ static struct test_case workqueue_testcase_table[] = { .args = (void *) workqueue_test_args, .n_args = (int)ARRAY_SIZE(workqueue_test_args), .arg_size =sizeof (struct workqueue_test_args_t ) + }, + { + .desc = "Workqueue parallel-for", + .func = __test__parallel_for, + .args = (void *) parallel_for_test_args, + .n_args = (int)ARRAY_SIZE(parallel_for_test_args), + .arg_size = sizeof(struct parallel_for_test_args_t) } };
tools/perf/util/workqueue/workqueue.c +125 -0
128 lines not on this page · raw message
tools/perf/util/workqueue/workqueue.h +7 -0
@@ -28,4 +28,11 @@ extern int flush_workqueue(struct workqueue_struct *wq);extern void init_work (struct work_struct *work); +/* parallel_for utility */ + +typedef void (*parallel_for_func_t )(int i,void *args); + +extern int parallel_for (struct workqueue_struct *wq,int from,int to,int work_size, +parallel_for_func_t func,void *args); +#endif /* __WORKQUEUE_WORKQUEUE_H */
Signature · 1 line
2.31.1
RFC PATCH 6/10 perf workqueue: introduce workqueue struct
In reply to Riccardo ManciniThis patch adds the workqueue definition, along with simple creation and destruction functions.
Furthermore, a simple subtest is added.
A workqueue is attached to a pool, on which it executes its workers. Next patches will introduce workers.
--- tools/perf/tests/workqueue.c | 92 +++++++++++++ tools/perf/util/workqueue/Build | 1 + tools/perf/util/workqueue/workqueue.c | 184 ++++++++++++++++++++++++++ tools/perf/util/workqueue/workqueue.h | 24 ++++ 4 files changed, 301 insertions(+) create mode 100644 tools/perf/util/workqueue/workqueue.c create mode 100644 tools/perf/util/workqueue/workqueue.h
tools/perf/tests/workqueue.c +92 -0
@@ -5,6 +5,7 @@#include "tests.h" #include "util/debug.h" #include "util/workqueue/threadpool.h" +#include "util/workqueue/workqueue.h" #define DUMMY_FACTOR 100000#define N_DUMMY_WORK_SIZES 7
@@ -13,6 +14,11 @@ struct threadpool_test_args_t {int pool_size; }; +struct workqueue_test_args_t { +int pool_size; +int n_work_items; +}; +struct test_task {struct task_struct task;int n_threads;
@@ -140,6 +146,58 @@ static int __test__threadpool(void *_args)return 0 ; } + +static int __workqueue__prepare (struct threadpool_struct **pool, +struct workqueue_struct **wq, +int pool_size) +{ +int ret; + + ret =__threadpool__prepare (pool, pool_size); +if (ret) +return ret; + + *wq =create_workqueue (*pool); +TEST_ASSERT_VAL ("workqueue creation failure" , *wq); +TEST_ASSERT_VAL ("workqueue wrong size" ,workqueue_nr_threads (*wq) == pool_size); +TEST_ASSERT_VAL ("threadpool is not executing" ,threadpool_is_busy (*pool)); + +return 0 ; +} + +static int __workqueue__teardown (struct threadpool_struct *pool, +struct workqueue_struct *wq) +{ +int ret; + + ret =destroy_workqueue (wq); +TEST_ASSERT_VAL ("workqueue detruction failure" , ret ==0 ); + + ret =__threadpool__teardown (pool); +if (ret) +return ret; + +return 0 ; +} + +static int __test__workqueue (void *_args) +{ +struct workqueue_test_args_t *args = _args; +struct threadpool_struct *pool; +struct workqueue_struct *wq; +int ret; + + ret =__workqueue__prepare (&pool, &wq, args->pool_size); +if (ret) +return ret; + + ret =__workqueue__teardown (pool, wq); +if (ret) +return ret; + +return 0 ; +} +static const struct threadpool_test_args_t threadpool_test_args[] = { { .pool_size =1
@@ -158,6 +216,33 @@ static const struct threadpool_test_args_t threadpool_test_args[] = { } }; +static const struct workqueue_test_args_t workqueue_test_args[] = { + { + .pool_size =1 , + .n_work_items =1 + }, + { + .pool_size =1 , + .n_work_items =10 + }, + { + .pool_size =2 , + .n_work_items =1 + }, + { + .pool_size =2 , + .n_work_items =100 + }, + { + .pool_size =16 , + .n_work_items =7 + }, + { + .pool_size =16 , + .n_work_items =2789 + } +}; +struct test_case {const char *desc;int (*func)(void *args);
@@ -173,6 +258,13 @@ static struct test_case workqueue_testcase_table[] = { .args = (void *) threadpool_test_args, .n_args = (int)ARRAY_SIZE(threadpool_test_args), .arg_size =sizeof (struct threadpool_test_args_t ) + }, + { + .desc = "Workqueue", + .func = __test__workqueue, + .args = (void *) workqueue_test_args, + .n_args = (int)ARRAY_SIZE(workqueue_test_args), + .arg_size = sizeof(struct workqueue_test_args_t) } };
tools/perf/util/workqueue/Build +1 -0
@@ -1 +1,2 @@ perf-y += threadpool.o +perf-y += workqueue.o
tools/perf/util/workqueue/workqueue.c new file +184 -0
184 lines not on this page · raw message
tools/perf/util/workqueue/workqueue.h new file +24 -0
@@ -0,0 +1,24 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __WORKQUEUE_WORKQUEUE_H +#define __WORKQUEUE_WORKQUEUE_H + +#include <stdlib.h> +#include <sys/types.h> +#include <linux/list.h> +#include "threadpool.h" + +struct work_struct ; +typedef void (*work_func_t )(struct work_struct *work); + +struct work_struct { +struct list_head entry; +work_func_t func; +}; + +struct workqueue_struct ; + +extern struct workqueue_struct *create_workqueue (struct threadpool_struct *pool); +extern int destroy_workqueue (struct workqueue_struct *wq); + +extern int workqueue_nr_threads (struct workqueue_struct *wq); +#endif /* __WORKQUEUE_WORKQUEUE_H */
Signature · 1 line
2.31.1
RFC PATCH 10/10 perf synthetic-events: use workqueue parallel_for
In reply to Riccardo ManciniTo generate synthetic events, perf has the option to use multiple threads. These threads are created manually using pthread_created.
This patch replaces the manual pthread_create with a workqueue, using the parallel_for utility.
Experimental results show that workqueue has a higher overhead, but this is repayed by the improved work balancing among threads.
Results of perf bench before and after are reported below:
Command: sudo ./perf bench internals synthesize -t Average synthesis time in usec is reported.
Laptop (dual core i7 w/ hyperthreading), avg num events ~14200:
N pthread (before) workqueue (after)
1 70714.400 +- 908.789 73306.000 +- 1597.868 2 77426.700 +- 2986.579 46782.300 +- 326.221 3 53176.300 +- 3405.635 41614.100 +- 239.827 4 50760.900 +- 702.623 41071.300 +- 230.200
VM (16 vCPUs over 16 core Intel Xeon E5-2630L v3), avg num events ~2760:
N pthread (before) workqueue (after)
1 30309.500 +- 578.283 34252.000 +- 839.474 2 23815.200 +- 1339.102 28487.200 +- 1423.481 3 20644.300 +- 311.573 19220.200 +- 1436.024 4 19091.500 +- 446.109 15048.600 +- 319.138 5 17574.000 +- 988.612 14938.500 +- 411.078 6 18908.900 +- 520.676 13997.600 +- 358.668 7 19275.700 +- 631.989 11371.400 +- 365.038 8 15671.200 +- 306.727 11964.800 +- 338.021 9 14660.900 +- 333.218 11762.800 +- 652.763 10 12490.200 +- 579.211 11832.300 +- 200.601 11 18052.900 +- 941.578 13166.900 +- 704.318 12 14253.600 +- 354.332 12012.000 +- 309.724 13 12219.000 +- 516.438 12023.800 +- 273.626 14 15896.600 +- 442.419 11764.600 +- 353.961 15 15087.200 +- 337.612 11942.600 +- 304.102 16 15368.700 +- 336.785 13625.200 +- 715.125
--- tools/perf/util/synthetic-events.c | 131 ++++++++++++----------------- 1 file changed, 56 insertions(+), 75 deletions(-)
tools/perf/util/synthetic-events.c +56 -75
@@ -41,6 +41,7 @@#include <sys/stat.h> #include <fcntl.h> #include <unistd.h> +#include "util/workqueue/workqueue.h" #define DEFAULT_PROC_MAP_PARSE_TIMEOUT 500
@@ -882,16 +883,13 @@ static int __perf_event__synthesize_threads(struct perf_tool *tool,perf_event__handler_t process, struct machine *machine, bool mmap_data, - struct dirent **dirent, - int start, - int num) + char *d_name) { union perf_event *comm_event, *mmap_event, *fork_event;union perf_event *namespaces_event;int err =-1 ;char *end;pid_t pid; -int i; comm_event =malloc (sizeof (comm_event->comm) + machine->id_hdr_size);if (comm_event ==NULL )
@@ -911,24 +909,22 @@ static int __perf_event__synthesize_threads(struct perf_tool *tool,if (namespaces_event ==NULL ) goto out_free_fork; -for (i = start; i < start + num; i++) { -if (!isdigit (dirent[i]->d_name[0 ])) -continue ; +if (!isdigit (d_name[0 ])) + goto out_free_namespaces; - pid = (pid_t )strtol (dirent[i]->d_name, &end,10 ); -/* only interested in proper numerical dirents */ -if (*end) -continue ; -/* -* We may race with exiting thread, so don't stop just because -* one thread couldn't be synthesized. -*/ -__event__synthesize_thread (comm_event, mmap_event, fork_event, - namespaces_event, pid,1 , process, - tool, machine, mmap_data); - } + pid = (pid_t )strtol (d_name, &end,10 ); +/* only interested in proper numerical dirents */ +if (*end) + goto out_free_namespaces; +/* +* We may race with exiting thread, so don't stop just because +* one thread couldn't be synthesized. +*/ +__event__synthesize_thread (comm_event, mmap_event, fork_event, + namespaces_event, pid,1 , process, + tool, machine, mmap_data); err =0 ; - +out_free_namespaces:free (namespaces_event); out_free_fork:free (fork_event);
@@ -946,19 +942,15 @@ struct synthesize_threads_arg {struct machine *machine;bool mmap_data;struct dirent **dirent; -int num; -int start; }; -static void *synthesize_threads_worker (void *arg) +static void synthesize_threads_worker (int i,void *arg) {struct synthesize_threads_arg *args = arg;__perf_event__synthesize_threads (args->tool, args->process, args->machine, args->mmap_data, - args->dirent, - args->start, args->num); -return NULL ; + args->dirent[i]->d_name); } int perf_event__synthesize_threads(struct perf_tool *tool,
@@ -967,15 +959,14 @@ int perf_event__synthesize_threads(struct perf_tool *tool,bool mmap_data, unsigned int nr_threads_synthesize) { - struct synthesize_threads_arg *args = NULL; -pthread_t *synthesize_threads =NULL ; + struct synthesize_threads_arg args;char proc_path[PATH_MAX ];struct dirent **dirent; -int num_per_thread; -int m, n, i, j; +int n, i;int thread_nr; -int base =0 ; -int err =-1 ; +int err =-1 , ret; +struct threadpool_struct *pool; +struct workqueue_struct *wq;if (machine__is_default_guest (machine))
@@ -992,54 +983,44 @@ int perf_event__synthesize_threads(struct perf_tool *tool, thread_nr = nr_threads_synthesize;if (thread_nr <=1 ) { - err =__perf_event__synthesize_threads (tool, process, - machine, mmap_data, - dirent, base, n); +for (i =0 ; i < n; i++) + err =__perf_event__synthesize_threads (tool, process, + machine, mmap_data, + dirent[i]->d_name); goto free_dirent; } -if (thread_nr > n) - thread_nr = n; - synthesize_threads =calloc (sizeof (pthread_t), thread_nr); -if (synthesize_threads ==NULL ) + pool =create_threadpool (thread_nr); +if (!pool) goto free_dirent; - args =calloc (sizeof (*args), thread_nr); -if (args ==NULL ) - goto free_threads; - - num_per_thread = n / thread_nr; - m = n % thread_nr; -for (i =0 ; i < thread_nr; i++) { - args[i].tool = tool; - args[i].process = process; - args[i].machine = machine; - args[i].mmap_data = mmap_data; - args[i].dirent = dirent; - } -for (i =0 ; i < m; i++) { - args[i].num = num_per_thread +1 ; - args[i].start = i * args[i].num; - } -if (i !=0 ) - base = args[i-1 ].start + args[i-1 ].num; -for (j = i; j < thread_nr; j++) { - args[j].num = num_per_thread; - args[j].start = base + (j - i) * args[i].num; - } - -for (i =0 ; i < thread_nr; i++) { -if (pthread_create (&synthesize_threads[i],NULL , - synthesize_threads_worker, &args[i])) - goto out_join; - } - err =0 ; -out_join: -for (i =0 ; i < thread_nr; i++) -pthread_join (synthesize_threads[i],NULL ); -free (args); -free_threads: -free (synthesize_threads); + err =start_threadpool (pool); +if (err) + goto free_pool; + + wq =create_workqueue (pool); +if (!wq) + goto stop_pool; + + args.tool = tool; + args.process = process; + args.machine = machine; + args.mmap_data = mmap_data; + args.dirent = dirent; + + ret =parallel_for (wq,0 , n,1 , synthesize_threads_worker, &args); +if (ret) + err = ret; + + ret =destroy_workqueue (wq); +if (ret) + err = ret; +stop_pool: + ret =stop_threadpool (pool); +if (ret) + err = ret; +free_pool: +destroy_threadpool (pool); free_dirent:for (i =0 ; i < n; i++)zfree (&dirent[i]);
Signature · 1 line
2.31.1
Arnaldo Carvalho de Melo
maintainerEm Tue, Jul 13, 2021 at 02:11:11PM +0200, Riccardo Mancini escreveu:
Quoted 35 lines
This patchset introduces a new utility library inside perf/util, which provides a work queue abstraction, which loosely follows the Kernel workqueue API.
The workqueue abstraction is made up by two components:
- threadpool: which takes care of managing a pool of threads. It is inspired by the prototype for threaded trace in perf-record from Alexey:https://lore.kernel.org/lkml/[email protected]/- workqueue: manages a shared queue and provides the workers implementation.
On top of the workqueue, a simple parallel-for utility is implemented which is then showcased in
synthetic-events.c, replacing the previous manual pthread-created threads.Through some experiments with perf bench, I can see how the new workqueue has a higher overhead compared to manual creation of threads, but is able to more effectively partition work among threads, yielding a better result with more threads.
Furthermore, the overhead could be configured by changing thework_size(currently 1), aka the number of dirents that are processed by a thread before grabbing a lock to get the new work item. I experimented with different sizes but, while bigger sizes reduce overhead as expected, they do not scale as well to more threads.I tried to keep the patchset as simple as possible, deferring possible improvements and features to future work.
Naming a few:
- in order to achieve a better performance, we could consider using work-stealing instead of a common queue.
- affinities in the thread pool, as in Alexey prototype for perf-record. Doing so would enable reusing the same threadpool for different purposes (evlist open, threaded trace, synthetic threads), avoiding having to spin up threads multiple times.
- resizable threadpool, e.g. for lazy spawining of threads.
Quoted 5 lines
@Arnaldo
Since I wanted the workqueue to provide a similar API to the Kernel's workqueue, I followed the naming style I found there, instead of the usualobject__methodstyle that is typically found in perf. Let me know if you'd like me to follow perf style instead.
You did the right thing, that is how we do with other kernel APIs, we use list_add(), rb_first(), bitmap_weight(), hash_del(), etc.
- Arnaldo
Quoted 39 lines
Thanks,
RiccardoRiccardo Mancini (10):
perf workqueue: threadpool creation and destruction perf tests: add test for workqueue
perf workqueue: add threadpool start and stop functions perf workqueue: add threadpool execute and wait functions perf workqueue: add sparse annotation header
perf workqueue: introduce workqueue struct
perf workqueue: implement worker thread and management perf workqueue: addqueue_workandflush_workqueuefunctions perf workqueue: add utility to execute a for loop in parallel perf synthetic-events: use workqueueparallel_fortools/perf/tests/Build | 1 + tools/perf/tests/builtin-test.c | 9 + tools/perf/tests/tests.h | 3 + tools/perf/tests/workqueue.c | 453 +++++++++++++++++ tools/perf/util/Build | 1 + tools/perf/util/synthetic-events.c | 131 +++-- tools/perf/util/workqueue/Build | 2 + tools/perf/util/workqueue/sparse.h | 21 + tools/perf/util/workqueue/threadpool.c | 516 ++++++++++++++++++++ tools/perf/util/workqueue/threadpool.h | 29 ++ tools/perf/util/workqueue/workqueue.c | 642 +++++++++++++++++++++++++ tools/perf/util/workqueue/workqueue.h | 38 ++ 12 files changed, 1771 insertions(+), 75 deletions(-) create mode 100644 tools/perf/tests/workqueue.c create mode 100644 tools/perf/util/workqueue/Build create mode 100644 tools/perf/util/workqueue/sparse.h create mode 100644 tools/perf/util/workqueue/threadpool.c create mode 100644 tools/perf/util/workqueue/threadpool.h create mode 100644 tools/perf/util/workqueue/workqueue.c create mode 100644 tools/perf/util/workqueue/workqueue.hSignature · 1 line
2.31.1
Signature · 1 line
- Arnaldo
Arnaldo Carvalho de Melo
maintainerRe: [RFC PATCH 01/10] perf workqueue: threadpool creation and destruction
In reply to Riccardo ManciniEm Tue, Jul 13, 2021 at 02:11:12PM +0200, Riccardo Mancini escreveu:
Quoted 60 lines
The workqueue library is made up by two components:
- threadpool: handles the lifetime of the threads
- workqueue: handles work distribution among the threadsThis first patch introduces the threadpool, starting from its creation and destruction functions.
Thread management is based on the prototype from Alexey:
https://lore.kernel.org/lkml/[email protected]/Each thread in the threadpool executes the same function (aka task) with a different argument tidx.
Threads use a pair of pipes to communicate with the main process. The threadpool is static (all threads will be spawned at the same time). Future work could include making it resizable and adding affinity support (as in Alexey prototype).Suggested-by Alexey Bayduraev <[email protected]>Signed-off-by Riccardo Mancini <[email protected]>--- tools/perf/util/Build | 1 + tools/perf/util/workqueue/Build | 1 + tools/perf/util/workqueue/threadpool.c | 175 +++++++++++++++++++++++++ tools/perf/util/workqueue/threadpool.h | 19 +++ 4 files changed, 196 insertions(+) create mode 100644 tools/perf/util/workqueue/Build create mode 100644 tools/perf/util/workqueue/threadpool.c create mode 100644 tools/perf/util/workqueue/threadpool.htools/perf/util/Build +1
-0@@ -180,6 +180,7 @@ perf-$(CONFIG_LIBBABELTRACE) += data-convert-bt.o perf-y += data-convert-json.o perf-y += scripting-engines/ +perf-y += workqueue/ perf-$(CONFIG_ZLIB) += zlib.o perf-$(CONFIG_LZMA) += lzma.otools/perf/util/workqueue/Build new file +1
-0@@ -0,0 +1 @@ +perf-y += threadpool.otools/perf/util/workqueue/threadpool.c new file +7
-0@@ -0,0 +1,175 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <stdlib.h> +#include <stdio.h> +#include <unistd.h> +#include <errno.h> +#include <string.h> +#include "debug.h"
tools/perf/util/workqueue/threadpool.c line 17
Quoted 10 lines
tools/perf/util/workqueue/threadpool.c new file +10
-0+#include "asm/bug.h" +#include "threadpool.h" + +enum threadpool_status { +THREADPOOL_STATUS__STOPPED ,/* no threads */ +THREADPOOL_STATUS__ERROR ,/* errors */ +THREADPOOL_STATUS__MAX +}; + +struct threadpool_struct {
Can this be just 'struct threadpool'? I think its descriptive enough:
Quoted 5 lines
tools/perf/util/workqueue/threadpool.c new file +5
-0+int nr_threads;/* number of threads in the pool */ +struct thread_struct *threads;/* array of threads in the pool */ +struct task_struct *current_task;/* current executing function */ +enum threadpool_status status;/* current status of the pool */ +};
tools/perf/util/workqueue/threadpool.c line 32
Quoted 10 lines
tools/perf/util/workqueue/threadpool.c new file +10
-0+ +struct thread_struct { +int idx;/* idx of thread in pool->threads */ +pid_t tid;/* tid of thread */ +struct threadpool_struct *pool;/* parent threadpool */ +struct { +int from[2 ];/* messages from thread (acks) */ +int to[2 ];/* messages to thread (commands) */ + } pipes; +};
This one, since we have already a 'struct thread' in tools/perf, to represent a PERF_RECORD_FORK, perhaps we can call it 'struct threadpool_entry'?
Quoted 6 lines
tools/perf/util/workqueue/threadpool.c new file +6
-0+ +/** +* init_pipes - initialize all pipes of @thread +*/ +static void init_pipes (struct thread_struct *thread) +{
tools/perf/util/workqueue/threadpool.c line 48
Quoted 10 lines
tools/perf/util/workqueue/threadpool.c new file +10
-0+ thread->pipes.from[0 ] =-1 ; + thread->pipes.from[1 ] =-1 ; + thread->pipes.to[0 ] =-1 ; + thread->pipes.to[1 ] =-1 ; +} + +/** +* open_pipes - open all pipes of @thread +*/ +static int open_pipes (struct thread_struct *thread)
Here please:
threadpool_entry__open_pipes()
Its longer, but helps with ctags/cscope navigation and we can go directly to it via:
:ta threadpool_entry__open_p<TAB>
While 'ta: open_pipes' may bo to various places where this idiom is used.
Quoted 37 lines
tools/perf/util/workqueue/threadpool.c new file +37
-0+{ +if (pipe (thread->pipes.from)) { +pr_err ("threadpool: failed to create comm pipe 'from': %s\n" , +strerror (errno)); +return -ENOMEM ; + } + +if (pipe (thread->pipes.to)) { +pr_err ("threadpool: failed to create comm pipe 'to': %s\n" , +strerror (errno)); +close (thread->pipes.from[0 ]); + thread->pipes.from[0 ] =-1 ; +close (thread->pipes.from[1 ]); + thread->pipes.from[1 ] =-1 ; +return -ENOMEM ; + } + +return 0 ; +} + +/** +* close_pipes - close all communication pipes of @thread +*/ +static void close_pipes (struct thread_struct *thread) +{ +if (thread->pipes.from[0 ] !=-1 ) { +close (thread->pipes.from[0 ]); + thread->pipes.from[0 ] =-1 ; + } +if (thread->pipes.from[1 ] !=-1 ) { +close (thread->pipes.from[1 ]); + thread->pipes.from[1 ] =-1 ; + } +if (thread->pipes.to[0 ] !=-1 ) { +close (thread->pipes.to[0 ]); + thread->pipes.to[0 ] =-1 ; + }
tools/perf/util/workqueue/threadpool.c line 95
Quoted 10 lines
tools/perf/util/workqueue/threadpool.c new file +10
-0+if (thread->pipes.to[1 ] !=-1 ) { +close (thread->pipes.to[1 ]); + thread->pipes.to[1 ] =-1 ; + } +} + +/** +* create_threadpool - create a fixed threadpool with @n_threads threads +*/ +struct threadpool_struct *create_threadpool (int n_threads)
Is this already something the kernel has and thus we should keep the naming? I couldn't find it in the kernel, so please name it:
struct threadpool *threadpool__new(int nthreads)
tools/perf/util/workqueue/threadpool.c
Quoted 7 lines
tools/perf/util/workqueue/threadpool.c new file +7
-0+{ +int ret, t; +struct threadpool_struct *pool =malloc (sizeof (*pool)); + +if (!pool) { +pr_err ("threadpool: cannot allocate pool: %s\n" , +strerror (errno));o
Humm, pr_err() at this level isn't appropriate, please make callers complain.
tools/perf/util/workqueue/threadpool.c line 108
Quoted 6 lines
tools/perf/util/workqueue/threadpool.c new file +6
-0+return NULL ; + } + +if (n_threads <=0 ) { +pr_err ("threadpool: invalid number of threads: %d\n" , + n_threads);
pr_debug()
Quoted 28 lines
tools/perf/util/workqueue/threadpool.c new file +28
-0+ goto out_free_pool; + } + + pool->nr_threads = n_threads; + pool->current_task =NULL ; + + pool->threads =malloc (n_threads *sizeof (*pool->threads)); +if (!pool->threads) { +pr_err ("threadpool: cannot allocate threads: %s\n" , +strerror (errno)); + goto out_free_pool; + } + +for (t =0 ; t < n_threads; t++) { + pool->threads[t].idx = t; + pool->threads[t].tid =-1 ; + pool->threads[t].pool = pool; +init_pipes (&pool->threads[t]); + } + +for (t =0 ; t < n_threads; t++) { + ret =open_pipes (&pool->threads[t]); +if (ret) + goto out_close_pipes; + } + + pool->status =THREADPOOL_STATUS__STOPPED ; +
tools/perf/util/workqueue/threadpool.c line 146
Quoted 10 lines
tools/perf/util/workqueue/threadpool.c new file +10
-0+return pool; + +out_close_pipes: +for (t =0 ; t < n_threads; t++) +close_pipes (&pool->threads[t]); + +free (pool->threads); +out_free_pool: +free (pool); +return NULL ;
Here we can use ERR_PTR()/PTR_ERR() to let the caller know what was the problem, i.e. we can ditch all the pr_err/pr_debug(), etc and instead have a threadpool__strerror(struct threadpool *pool, int err) like we have for 'struct evsel', please take a look at evsel__open_strerror().
tools/perf/util/workqueue/threadpool.c line 152
Quoted 6 lines
tools/perf/util/workqueue/threadpool.c new file +6
-0+} + +/** +* destroy_threadpool - free the @pool and all its resources +*/ +void destroy_threadpool (struct threadpool_struct *pool)
void threadpool__delete(struct threadpool *pool)
tools/perf/util/workqueue/threadpool.c line 163
Quoted 11 lines
tools/perf/util/workqueue/threadpool.c new file +11
-0+{ +int t; + +if (!pool) +return ; + +WARN_ON (pool->status !=THREADPOOL_STATUS__STOPPED + && pool->status !=THREADPOOL_STATUS__ERROR ); + +for (t =0 ; t < pool->nr_threads; t++) +close_pipes (&pool->threads[t]);
reset pool->threads[t] to -1
tools/perf/util/workqueue/threadpool.c line 165
Quoted 2 lines
tools/perf/util/workqueue/threadpool.c new file +2
-0+ +free (pool->threads);
zfree
tools/perf/util/workqueue/threadpool.c line 172
Quoted 7 lines
tools/perf/util/workqueue/threadpool.c new file +7
-0+free (pool); +} + +/** +* threadpool_size - get number of threads in the threadpool +*/ +int threadpool_size (struct threadpool_struct *pool)
threadpool__size()
Quoted 31 lines
tools/perf/util/workqueue/threadpool.c new file +3
-0+{ +return pool->nr_threads; +}tools/perf/util/workqueue/threadpool.h new file +19
-0@@ -0,0 +1,19 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __WORKQUEUE_THREADPOOL_H +#define __WORKQUEUE_THREADPOOL_H + +struct threadpool_struct ; +struct task_struct ; + +typedef void (*task_func_t )(int tidx,struct task_struct *task); + +struct task_struct { +task_func_t fn; +}; + +extern struct threadpool_struct *create_threadpool (int n_threads); +extern void destroy_threadpool (struct threadpool_struct *pool); + +extern int threadpool_size (struct threadpool_struct *pool); + +#endif /* __WORKQUEUE_THREADPOOL_H */ Signature · 1 line
2.31.1
Signature · 1 line
- Arnaldo
Arnaldo Carvalho de Melo
maintainerRe: [RFC PATCH 02/10] perf tests: add test for workqueue
In reply to Riccardo ManciniEm Tue, Jul 13, 2021 at 02:11:13PM +0200, Riccardo Mancini escreveu:
Quoted 92 lines
It will have subtests testing threadpool and workqueue separately. This patch only introduces the first subtest, checking that the threadpool is correctly created and destructed.
This test will be expanded when new functions are added in next patches.Signed-off-by Riccardo Mancini <[email protected]>--- tools/perf/tests/Build | 1 + tools/perf/tests/builtin-test.c | 9 +++ tools/perf/tests/tests.h | 3 + tools/perf/tests/workqueue.c | 113 ++++++++++++++++++++++++++++++++ 4 files changed, 126 insertions(+) create mode 100644 tools/perf/tests/workqueue.ctools/perf/tests/Build +1
-0@@ -64,6 +64,7 @@ perf-y += parse-metric.o perf-y += pe-file-parsing.o perf-y += expand-cgroup.o perf-y += perf-time-to-tsc.o +perf-y += workqueue.o $(OUTPUT)tests/llvm-src-base.c: tests/bpf-script-example.c tests/Build $(call rule_mkdir)tools/perf/tests/builtin-test.c +9
-0@@ -360,6 +360,15 @@ static struct test generic_tests[] = { .func = test__perf_time_to_tsc, .is_supported = test__tsc_is_supported, }, + { + .desc = "Test workqueue lib", + .func = test__workqueue, + .subtest = { + .skip_if_fail = false, + .get_nr = test__workqueue_subtest_get_nr, + .get_desc = test__workqueue_subtest_get_desc, + } + }, { .func = NULL, },tools/perf/tests/tests.h +3
-0@@ -127,6 +127,9 @@ int test__parse_metric(struct test *test, int subtest);int test__pe_file_parsing (struct test *test,int subtest);int test__expand_cgroup_events (struct test *test,int subtest);int test__perf_time_to_tsc (struct test *test,int subtest); +int test__workqueue (struct test *test,int subtest); +const char *test__workqueue_subtest_get_desc (int subtest); +int test__workqueue_subtest_get_nr (void );bool test__bp_signal_is_supported (void );bool test__bp_account_is_supported (void );tools/perf/tests/workqueue.c new file +25
-0@@ -0,0 +1,113 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <linux/kernel.h> +#include "tests.h" +#include "util/debug.h" +#include "util/workqueue/threadpool.h" + +struct threadpool_test_args_t { +int pool_size; +}; + +static int __threadpool__prepare (struct threadpool_struct **pool,int pool_size) +{ + *pool =create_threadpool (pool_size); +TEST_ASSERT_VAL ("threadpool creation failure" , *pool !=NULL ); +TEST_ASSERT_VAL ("threadpool size is wrong" , +threadpool_size (*pool) == pool_size); + +return 0 ; +} + +static int __threadpool__teardown (struct threadpool_struct *pool) +{ +destroy_threadpool (pool); + +return 0 ;
tools/perf/tests/workqueue.c line 35
Quoted 10 lines
tools/perf/tests/workqueue.c new file +10
-0+} + + +static int __test__threadpool (void *_args) +{ +struct threadpool_test_args_t *args = _args; +struct threadpool_struct *pool; +int ret; + + ret =__threadpool__prepare (&pool, args->pool_size);
Turn the last three lines into one;
int ret = __threadpool__prepare(&pool, args->pool_size);
tools/perf/tests/workqueue.c line 43
Quoted 8 lines
tools/perf/tests/workqueue.c new file +8
-0+if (ret) +return ret; + + ret =__threadpool__teardown (pool); +if (ret) +return ret; + +return 0 ;
Humm, will you add something here in the following csets? Otherwise turn these 5 lines into one:
return __threadpool__teardown(pool);
Quoted 73 lines
tools/perf/tests/workqueue.c new file +70
-0+} + +static const struct threadpool_test_args_t threadpool_test_args[] = { + { + .pool_size =1 + }, + { + .pool_size =2 + }, + { + .pool_size =4 + }, + { + .pool_size =8 + }, + { + .pool_size =16 + } +}; + +struct test_case { +const char *desc; +int (*func)(void *args); +void *args; +int n_args; +int arg_size; +}; + +static struct test_case workqueue_testcase_table[] = { + { + .desc ="Threadpool" , + .func = __test__threadpool, + .args = (void *) threadpool_test_args, + .n_args = (int )ARRAY_SIZE (threadpool_test_args), + .arg_size =sizeof (struct threadpool_test_args_t ) + } +}; + + +int test__workqueue (struct test *test__maybe_unused ,int i) +{ +int j, ret =0 ; +struct test_case *tc; + +if (i <0 || i >= (int )ARRAY_SIZE (workqueue_testcase_table)) +return -1 ; + + tc = &workqueue_testcase_table[i]; + +for (j =0 ; j < tc->n_args; j++) { + ret = tc->func (tc->args + (j*tc->arg_size)); +if (ret) +return ret; + } + +return 0 ; +} + + +int test__workqueue_subtest_get_nr (void ) +{ +return (int )ARRAY_SIZE (workqueue_testcase_table); +} + +const char *test__workqueue_subtest_get_desc (int i) +{ +if (i <0 || i >= (int )ARRAY_SIZE (workqueue_testcase_table)) +return NULL ; +return workqueue_testcase_table[i].desc; +}Signature · 1 line
2.31.1
Signature · 1 line
- Arnaldo
Arnaldo Carvalho de Melo
maintainerRe: [RFC PATCH 03/10] perf workqueue: add threadpool start and stop functions
In reply to Riccardo ManciniEm Tue, Jul 13, 2021 at 02:11:14PM +0200, Riccardo Mancini escreveu:
Quoted 33 lines
This patch adds the start and stop functions, alongside the thread function.
Each thread will run until a stop signal is received. Furthermore, start and stop are added to the test.Thread management is based on the prototype from Alexey:
https://lore.kernel.org/lkml/[email protected]/Suggested-by Alexey Bayduraev <[email protected]>Signed-off-by Riccardo Mancini <[email protected]>--- tools/perf/tests/workqueue.c | 13 ++ tools/perf/util/workqueue/threadpool.c | 238 +++++++++++++++++++++++++ tools/perf/util/workqueue/threadpool.h | 5 + 3 files changed, 256 insertions(+)tools/perf/tests/workqueue.c +4
-0@@ -10,16 +10,29 @@ struct threadpool_test_args_t {static int __threadpool__prepare (struct threadpool_struct **pool,int pool_size) { +int ret; + *pool =create_threadpool (pool_size);TEST_ASSERT_VAL ("threadpool creation failure" , *pool !=NULL );TEST_ASSERT_VAL ("threadpool size is wrong" ,threadpool_size (*pool) == pool_size); + ret =start_threadpool (*pool); +TEST_ASSERT_VAL ("threadpool start failure" , ret ==0 );
tools/perf/tests/workqueue.c struct threadpool_test_args_t { · line 31
Quoted 10 lines
tools/perf/tests/workqueue.c +5
-0+TEST_ASSERT_VAL ("threadpool is not ready" ,threadpool_is_ready (*pool)); +return 0 ; }static int __threadpool__teardown (struct threadpool_struct *pool) { +int ret; + + ret =stop_threadpool (pool);
int ret = stop_threadpool(pool);
Quoted 18 lines
tools/perf/tests/workqueue.c +4
-0+TEST_ASSERT_VAL ("threadpool start failure" , ret ==0 ); +TEST_ASSERT_VAL ("stopped threadpool is ready" , + !threadpool_is_ready (pool)); +destroy_threadpool (pool);return 0 ;tools/perf/util/workqueue/threadpool.c +3
-0@@ -4,12 +4,23 @@#include <unistd.h> #include <errno.h> #include <string.h> +#include <pthread.h> +#include <signal.h> +#include <syscall.h>
tools/perf/util/workqueue/threadpool.c line 19
Quoted 10 lines
tools/perf/util/workqueue/threadpool.c +6
-0#include "debug.h" #include "asm/bug.h" #include "threadpool.h" +#ifndef HAVE_GETTID +static inline pid_t gettid (void ) +{ +return (pid_t )syscall (__NR_gettid); +} +#endif
Isn't this defined elsewhere? Yeah, when we decide to move it to tools/lib/workqueue/ we'll need it, but for now, reduce patch size.
Quoted 33 lines
tools/perf/util/workqueue/threadpool.c +17
-0enum threadpool_status {THREADPOOL_STATUS__STOPPED ,/* no threads */ +THREADPOOL_STATUS__READY ,/* threads are ready but idle */ THREADPOOL_STATUS__ERROR ,/* errors */ THREADPOOL_STATUS__MAX };@@ -31,6 +42,21 @@ struct thread_struct { } pipes; }; +enum thread_msg { +THREAD_MSG__UNDEFINED =0 , +THREAD_MSG__ACK ,/* from th: create and exit ack */ +THREAD_MSG__WAKE ,/* to th: wake up */ +THREAD_MSG__STOP ,/* to th: exit */ +THREAD_MSG__MAX +}; + +static const char *const thread_msg_tags[] = { +"undefined" , +"ack" , +"wake" , +"stop" +}; +/** * init_pipes - initialize all pipes of @thread */ @@ -89,6 +115,113 @@ static void close_pipes(struct thread_struct *thread) } } +/**
tools/perf/util/workqueue/threadpool.c static void close_pipes(struct thread_struct *thread) · line 128
Quoted 10 lines
tools/perf/util/workqueue/threadpool.c +10
-0+* wait_thread - receive ack from thread +* +* NB: call only from main thread! +*/ +static int wait_thread (struct thread_struct *thread) +{ +int res; +enum thread_msg msg =THREAD_MSG__UNDEFINED ; + + res =read (thread->pipes.from[0 ], &msg,sizeof (msg));
int res = read(thread->pipes.from[0], &msg, sizeof(msg));
Quoted 37 lines
tools/perf/util/workqueue/threadpool.c +37
-0+if (res <0 ) { +pr_err ("threadpool: failed to recv msg from tid=%d: %s\n" , + thread->tid,strerror (errno)); +return -1 ; + } +if (msg !=THREAD_MSG__ACK ) { +pr_err ("threadpool: received unexpected msg from tid=%d: %s\n" , + thread->tid, thread_msg_tags[msg]); +return -1 ; + } + +pr_debug2 ("threadpool: received ack from tid=%d\n" , thread->tid); + +return 0 ; +} + +/** +* terminate_thread - send stop signal to thread and wait for ack +* +* NB: call only from main thread! +*/ +static int terminate_thread (struct thread_struct *thread) +{ +int res; +enum thread_msg msg =THREAD_MSG__STOP ; + + res =write (thread->pipes.to[1 ], &msg,sizeof (msg)); +if (res <0 ) { +pr_err ("threadpool: error sending stop msg to tid=%d: %s\n" , + thread->tid,strerror (errno)); +return res; + } + + res =wait_thread (thread); + +return res; +}
tools/perf/util/workqueue/threadpool.c static void close_pipes(struct thread_struct *thread) · line 175
Quoted 10 lines
tools/perf/util/workqueue/threadpool.c +10
-0+ +/** +* threadpool_thread - function running on thread +* +* This function waits for a signal from main thread to start executing +* a task. +* On completion, it will go back to sleep, waiting for another signal. +* Signals are delivered through pipes. +*/ +static void *threadpool_thread (void *args)
threadpool_function()
ETOMANY 'thread' in a name.
Quoted 106 lines
tools/perf/util/workqueue/threadpool.c +99
-0+{ +struct thread_struct *thread = (struct thread_struct *) args; +enum thread_msg msg; +int err; + + thread->tid =gettid (); + +pr_debug2 ("threadpool[%d]: started\n" , thread->tid); + +for (;;) { + msg =THREAD_MSG__ACK ; + err =write (thread->pipes.from[1 ], &msg,sizeof (msg)); +if (err ==-1 ) { +pr_err ("threadpool[%d]: failed to send ack: %s\n" , + thread->tid,strerror (errno)); +break ; + } + + msg =THREAD_MSG__UNDEFINED ; + err =read (thread->pipes.to[0 ], &msg,sizeof (msg)); +if (err <0 ) { +pr_err ("threadpool[%d]: error receiving msg: %s\n" , + thread->tid,strerror (errno)); +break ; + } + +if (msg !=THREAD_MSG__WAKE && msg !=THREAD_MSG__STOP ) { +pr_err ("threadpool[%d]: received unexpected msg: %s\n" , + thread->tid, thread_msg_tags[msg]); +break ; + } + +if (msg ==THREAD_MSG__STOP ) +break ; + } + +pr_debug2 ("threadpool[%d]: exit\n" , thread->tid); + + msg =THREAD_MSG__ACK ; + err =write (thread->pipes.from[1 ], &msg,sizeof (msg)); +if (err ==-1 ) { +pr_err ("threadpool[%d]: failed to send ack: %s\n" , + thread->tid,strerror (errno)); +return NULL ; + } + +return NULL ; +} +/** * create_threadpool - create a fixed threadpool with @n_threads threads */ @@ -173,3 +306,108 @@ int threadpool_size(struct threadpool_struct *pool) {return pool->nr_threads; } + +/** +* __start_threadpool - start all threads in the pool. +* +* This function does not change @pool->status. +*/ +static int __start_threadpool (struct threadpool_struct *pool) +{ +int t, tt, ret =0 , nr_threads = pool->nr_threads; +sigset_t full, mask; +pthread_t handle; +pthread_attr_t attrs; + +sigfillset (&full); +if (sigprocmask (SIG_SETMASK , &full, &mask)) { +pr_err ("Failed to block signals on threads start: %s\n" , +strerror (errno)); +return -1 ; + } + +pthread_attr_init (&attrs); +pthread_attr_setdetachstate (&attrs,PTHREAD_CREATE_DETACHED ); + +for (t =0 ; t < nr_threads; t++) { +struct thread_struct *thread = &pool->threads[t]; + +if (pthread_create (&handle, &attrs, threadpool_thread, thread)) { +for (tt =1 ; tt < t; tt++) +terminate_thread (thread); +pr_err ("Failed to start threads: %s\n" ,strerror (errno)); + ret =-1 ; + goto out_free_attr; + } + +if (wait_thread (thread)) { +for (tt =1 ; tt <= t; tt++) +terminate_thread (thread); + ret =-1 ; + goto out_free_attr; + } + } + +out_free_attr: +pthread_attr_destroy (&attrs); + +if (sigprocmask (SIG_SETMASK , &mask,NULL )) { +pr_err ("Failed to unblock signals on threads start: %s\n" , +strerror (errno)); + ret =-1 ; + }
tools/perf/util/workqueue/threadpool.c int threadpool_size(struct threadpool_struct *pool) · line 368
Quoted 10 lines
tools/perf/util/workqueue/threadpool.c +10
-0+ +return ret; +} + +/** +* start_threadpool - start all threads in the pool. +* +* The function blocks until all threads are up and running. +*/ +int start_threadpool (struct threadpool_struct *pool)
int threadpool__start(struct threadpool *pool)
Quoted 9 lines
tools/perf/util/workqueue/threadpool.c +9
-0+{ +int err; + +if (pool->status !=THREADPOOL_STATUS__STOPPED ) { +pr_err ("threadpool: starting not stopped pool\n" ); +return -1 ; + } + + err =__start_threadpool (pool);
tools/perf/util/workqueue/threadpool.c int threadpool_size(struct threadpool_struct *pool) · line 387
Quoted 10 lines
tools/perf/util/workqueue/threadpool.c +10
-0+ pool->status = err ?THREADPOOL_STATUS__ERROR :THREADPOOL_STATUS__READY ; +return err; +} + +/** +* stop_threadpool - stop all threads in the pool. +* +* This function blocks waiting for ack from all threads. +*/ +int stop_threadpool (struct threadpool_struct *pool)
int threadpool__stop(struct threadpool *pool)
Quoted 13 lines
tools/perf/util/workqueue/threadpool.c +13
-0+{ +int t, ret, err =0 ; + +if (pool->status !=THREADPOOL_STATUS__READY ) { +pr_err ("threadpool: stopping not ready pool\n" ); +return -1 ; + } + +for (t =0 ; t < pool->nr_threads; t++) { + ret =terminate_thread (&pool->threads[t]); +if (ret && !err) + err =-1 ; + }
tools/perf/util/workqueue/threadpool.c int threadpool_size(struct threadpool_struct *pool) · line 410
Quoted 10 lines
tools/perf/util/workqueue/threadpool.c +10
-0+ + pool->status = err ?THREADPOOL_STATUS__ERROR :THREADPOOL_STATUS__STOPPED ; + +return err; +} + +/** +* threadpool_is_ready - check if the threads are running +*/ +bool threadpool_is_ready (struct threadpool_struct *pool)
bool threadpool__is_ready(struct threadpool *pool)
Quoted 22 lines
tools/perf/util/workqueue/threadpool.c +3
-0+{ +return pool->status ==THREADPOOL_STATUS__READY ; +}tools/perf/util/workqueue/threadpool.h +5
-0@@ -14,6 +14,11 @@ struct task_struct {extern struct threadpool_struct *create_threadpool (int n_threads);extern void destroy_threadpool (struct threadpool_struct *pool); +extern int start_threadpool (struct threadpool_struct *pool); +extern int stop_threadpool (struct threadpool_struct *pool); +extern int threadpool_size (struct threadpool_struct *pool); +extern bool threadpool_is_ready (struct threadpool_struct *pool); +#endif /* __WORKQUEUE_THREADPOOL_H */ Signature · 1 line
2.31.1
Signature · 1 line
- Arnaldo
Arnaldo Carvalho de Melo
maintainerRe: [RFC PATCH 06/10] perf workqueue: introduce workqueue struct
In reply to Riccardo ManciniEm Tue, Jul 13, 2021 at 02:11:17PM +0200, Riccardo Mancini escreveu:
Quoted 240 lines
This patch adds the workqueue definition, along with simple creation and destruction functions.
Furthermore, a simple subtest is added.A workqueue is attached to a pool, on which it executes its workers. Next patches will introduce workers.
Signed-off-by Riccardo Mancini <[email protected]>--- tools/perf/tests/workqueue.c | 92 +++++++++++++ tools/perf/util/workqueue/Build | 1 + tools/perf/util/workqueue/workqueue.c | 184 ++++++++++++++++++++++++++ tools/perf/util/workqueue/workqueue.h | 24 ++++ 4 files changed, 301 insertions(+) create mode 100644 tools/perf/util/workqueue/workqueue.c create mode 100644 tools/perf/util/workqueue/workqueue.htools/perf/tests/workqueue.c +92
-0@@ -5,6 +5,7 @@#include "tests.h" #include "util/debug.h" #include "util/workqueue/threadpool.h" +#include "util/workqueue/workqueue.h" #define DUMMY_FACTOR 100000#define N_DUMMY_WORK_SIZES 7@@ -13,6 +14,11 @@ struct threadpool_test_args_t {int pool_size; }; +struct workqueue_test_args_t { +int pool_size; +int n_work_items; +}; +struct test_task {struct task_struct task;int n_threads;@@ -140,6 +146,58 @@ static int __test__threadpool(void *_args)return 0 ; } + +static int __workqueue__prepare (struct threadpool_struct **pool, +struct workqueue_struct **wq, +int pool_size) +{ +int ret; + + ret =__threadpool__prepare (pool, pool_size); +if (ret) +return ret; + + *wq =create_workqueue (*pool); +TEST_ASSERT_VAL ("workqueue creation failure" , *wq); +TEST_ASSERT_VAL ("workqueue wrong size" ,workqueue_nr_threads (*wq) == pool_size); +TEST_ASSERT_VAL ("threadpool is not executing" ,threadpool_is_busy (*pool)); + +return 0 ; +} + +static int __workqueue__teardown (struct threadpool_struct *pool, +struct workqueue_struct *wq) +{ +int ret; + + ret =destroy_workqueue (wq); +TEST_ASSERT_VAL ("workqueue detruction failure" , ret ==0 ); + + ret =__threadpool__teardown (pool); +if (ret) +return ret; + +return 0 ; +} + +static int __test__workqueue (void *_args) +{ +struct workqueue_test_args_t *args = _args; +struct threadpool_struct *pool; +struct workqueue_struct *wq; +int ret; + + ret =__workqueue__prepare (&pool, &wq, args->pool_size); +if (ret) +return ret; + + ret =__workqueue__teardown (pool, wq); +if (ret) +return ret; + +return 0 ; +} +static const struct threadpool_test_args_t threadpool_test_args[] = { { .pool_size =1 @@ -158,6 +216,33 @@ static const struct threadpool_test_args_t threadpool_test_args[] = { } }; +static const struct workqueue_test_args_t workqueue_test_args[] = { + { + .pool_size =1 , + .n_work_items =1 + }, + { + .pool_size =1 , + .n_work_items =10 + }, + { + .pool_size =2 , + .n_work_items =1 + }, + { + .pool_size =2 , + .n_work_items =100 + }, + { + .pool_size =16 , + .n_work_items =7 + }, + { + .pool_size =16 , + .n_work_items =2789 + } +}; +struct test_case {const char *desc;int (*func)(void *args);@@ -173,6 +258,13 @@ static struct test_case workqueue_testcase_table[] = { .args = (void *) threadpool_test_args, .n_args = (int)ARRAY_SIZE(threadpool_test_args), .arg_size =sizeof (struct threadpool_test_args_t ) + }, + { + .desc = "Workqueue", + .func = __test__workqueue, + .args = (void *) workqueue_test_args, + .n_args = (int)ARRAY_SIZE(workqueue_test_args), + .arg_size = sizeof(struct workqueue_test_args_t) } };tools/perf/util/workqueue/Build +1
-0@@ -1 +1,2 @@ perf-y += threadpool.o +perf-y += workqueue.otools/perf/util/workqueue/workqueue.c new file +79
-0@@ -0,0 +1,184 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <stdlib.h> +#include <stdio.h> +#include <unistd.h> +#include <errno.h> +#include <string.h> +#include <pthread.h> +#include <linux/list.h> +#include "debug.h" +#include "workqueue.h" + +enum workqueue_status { +WORKQUEUE_STATUS__READY ,/* wq is ready to receive work */ +WORKQUEUE_STATUS__ERROR , +WORKQUEUE_STATUS__MAX +}; + +struct workqueue_struct { +pthread_mutex_t lock;/* locking of the thread_pool */ +pthread_cond_t idle_cond;/* all workers are idle cond */ +struct threadpool_struct *pool;/* underlying pool */ +struct task_struct task;/* threadpool task */ +struct list_head busy_list;/* busy workers */ +struct list_head idle_list;/* idle workers */ +struct list_head pending;/* pending work items */ +int msg_pipe[2 ];/* main thread comm pipes */ +enum workqueue_status status; +}; + +/** +* worker_thread - worker function executed on threadpool +*/ +static void worker_thread (int tidx,struct task_struct *task) +{ + struct workqueue_struct *wq = container_of(task, struct workqueue_struct, task); + +pr_debug ("hi from worker %d. Pool is in status %d\n" , tidx, wq->status); +} + +/** +* attach_threadpool_to_workqueue - start @wq workers on @pool +*/ +static int attach_threadpool_to_workqueue (struct workqueue_struct *wq, +struct threadpool_struct *pool) +{ +int err; + +if (!threadpool_is_ready (pool)) { +pr_err ("workqueue: cannot attach to pool: pool is not ready\n" ); +return -1 ; + } + + wq->pool = pool; + + err =execute_in_threadpool (pool, &wq->task); +if (err) +return -1 ; + +return 0 ; +} + +/** +* detach_threadpool_from_workqueue - stop @wq workers on @pool +*/ +static int detach_threadpool_from_workqueue (struct workqueue_struct *wq) +{ +int ret, err =0 ; + +if (wq->status !=WORKQUEUE_STATUS__READY ) { +pr_err ("workqueue: cannot attach to pool: wq is not ready\n" ); +return -1 ; + } + + ret =wait_threadpool (wq->pool); +if (ret) { +pr_err ("workqueue: error waiting threadpool\n" ); + err =-1 ; + } +
tools/perf/util/workqueue/workqueue.c line 89
Quoted 10 lines
tools/perf/util/workqueue/workqueue.c new file +10
-0+ wq->pool =NULL ; +return err; +} + +/** +* create_workqueue - create a workqueue associated to @pool +* +* Only one workqueue can execute on a pool at a time. +*/ +struct workqueue_struct *create_workqueue (struct threadpool_struct *pool)
I wonder if we should use the exact same kernel signature and not pass a threadpool, essentially having just one threadpool in tools/perf/ that is used by create_workqueue(void)?
Quoted 128 lines
tools/perf/util/workqueue/workqueue.c new file +95
-0+{ +int err; +struct workqueue_struct *wq =malloc (sizeof (struct workqueue_struct )); + + + err =pthread_mutex_init (&wq->lock,NULL ); +if (err) + goto out_free_wq; + + err =pthread_cond_init (&wq->idle_cond,NULL ); +if (err) + goto out_destroy_mutex; + + wq->pool =NULL ; +INIT_LIST_HEAD (&wq->busy_list); +INIT_LIST_HEAD (&wq->idle_list); + +INIT_LIST_HEAD (&wq->pending); + + err =pipe (wq->msg_pipe); +if (err) + goto out_destroy_cond; + + wq->task.fn = worker_thread; + + err =attach_threadpool_to_workqueue (wq, pool); +if (err) + goto out_destroy_cond; + + wq->status =WORKQUEUE_STATUS__READY ; + +return wq; + +out_destroy_cond: +pthread_cond_destroy (&wq->idle_cond); +out_destroy_mutex: +pthread_mutex_destroy (&wq->lock); +out_free_wq: +free (wq); +return NULL ; +} + +/** +* destroy_workqueue - stop @wq workers and destroy @wq +*/ +int destroy_workqueue (struct workqueue_struct *wq) +{ +int err =0 , ret; + + ret =detach_threadpool_from_workqueue (wq); +if (ret) { +pr_err ("workqueue: error detaching from threadpool.\n" ); + err =-1 ; + } + + ret =pthread_mutex_destroy (&wq->lock); +if (ret) { + err =-1 ; +pr_err ("workqueue: error pthread_mutex_destroy: %s\n" , +strerror (errno)); + } + + ret =pthread_cond_destroy (&wq->idle_cond); +if (ret) { + err =-1 ; +pr_err ("workqueue: error pthread_cond_destroy: %s\n" , +strerror (errno)); + } + + ret =close (wq->msg_pipe[0 ]); +if (ret) { + err =-1 ; +pr_err ("workqueue: error close msg_pipe[0]: %s\n" , +strerror (errno)); + } + + ret =close (wq->msg_pipe[1 ]); +if (ret) { + err =-1 ; +pr_err ("workqueue: error close msg_pipe[1]: %s\n" , +strerror (errno)); + } + +free (wq); + +return err; +} + +/** +* workqueue_nr_threads - get size of threadpool underlying @wq +*/ +int workqueue_nr_threads (struct workqueue_struct *wq) +{ +return threadpool_size (wq->pool); +}tools/perf/util/workqueue/workqueue.h new file +24
-0@@ -0,0 +1,24 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __WORKQUEUE_WORKQUEUE_H +#define __WORKQUEUE_WORKQUEUE_H + +#include <stdlib.h> +#include <sys/types.h> +#include <linux/list.h> +#include "threadpool.h" + +struct work_struct ; +typedef void (*work_func_t )(struct work_struct *work); + +struct work_struct { +struct list_head entry; +work_func_t func; +}; + +struct workqueue_struct ; + +extern struct workqueue_struct *create_workqueue (struct threadpool_struct *pool); +extern int destroy_workqueue (struct workqueue_struct *wq); + +extern int workqueue_nr_threads (struct workqueue_struct *wq); +#endif /* __WORKQUEUE_WORKQUEUE_H */ Signature · 1 line
2.31.1
Signature · 1 line
- Arnaldo
Re: [RFC PATCH 01/10] perf workqueue: threadpool creation and destruction
In reply to Arnaldo Carvalho de MeloHi Arnaldo,
thanks for reviewing the patch!
On Wed, 2021-07-14 at 11:16 -0300, Arnaldo Carvalho de Melo wrote:
<SNIP>
Quoted 10 lines
tools/perf/util/workqueue/threadpool.c line 17
Quoted 8 lines
tools/perf/util/workqueue/threadpool.c new file +8
-0+ +enum threadpool_status { + THREADPOOL_STATUS__STOPPED, /* no threads */ + THREADPOOL_STATUS__ERROR, /* errors */ + THREADPOOL_STATUS__MAX +}; + +struct threadpool_struct {Can this be just 'struct threadpool'? I think its descriptive enough:
I agree, but I wanted to keep the naming consistent between workqueue.c and threadpool.c.
Quoted 27 lines
Quoted 16 lines
+ int nr_threads; /* number of threads in the pool */
+ struct thread_struct *threads; /* array of threads in the pool */
+ struct task_struct *current_task; /* current executing function
*/
+ enum threadpool_status status; /* current status of the pool
*/tools/perf/util/workqueue/threadpool.c new file +4
-0+}; + +struct thread_struct { + int idx; /* idx of thread in pool-Quoted 1 line
threads */
Quoted 10 lines
+ pid_t tid; /* tid of thread */ + struct threadpool_struct *pool; /* parent threadpool */ + struct {
+ int from[2]; /* messages from thread (acks)
*/
+ int to[2]; /* messages to thread (commands) */
+ } pipes;
+};
Quoted 3 lines
This one, since we have already a 'struct thread' in
tools/perf, to represent aPERF_RECORD_FORK, perhaps we can call it 'structthreadpool_entry'?
Agreed.
Quoted 22 lines
Quoted 7 lines
tools/perf/util/workqueue/threadpool.c new file +6
-0+ +/** +* init_pipes - initialize all pipes of @thread +*/ +static void init_pipes (struct thread_struct *thread) +{tools/perf/util/workqueue/threadpool.c line 48
Quoted 10 lines
tools/perf/util/workqueue/threadpool.c new file +10
-0+ thread->pipes.from[0] = -1; + thread->pipes.from[1] = -1; + thread->pipes.to[0] = -1; + thread->pipes.to[1] = -1; +} + +/** +* open_pipes - open all pipes of @thread +*/ +static int open_pipes (struct thread_struct *thread)Here please:
threadpool_entry__open_pipes()
Quoted 7 lines
Its longer, but helps with ctags/cscope navigation and we can go directly to it via:
:ta threadpool_entry__open_p<TAB>
While 'ta:
open_pipes' may bo to various places where this idiom is used.
Agreed.
<SNIP>
Quoted 10 lines
tools/perf/util/workqueue/threadpool.c line 95
Quoted 4 lines
tools/perf/util/workqueue/threadpool.c new file +4
-0+/** +* create_threadpool - create a fixed threadpool with @n_threads threads +*/ +struct threadpool_struct *create_threadpool (int n_threads)Is this already something the kernel has and thus we should keep the naming? I couldn't find it in the kernel, so please name it:
struct threadpool *threadpool__new(int nthreads)
As before, I did this to keep consistency with workqueue.
Since this threadpool+workqueue can be a standalone library, I preferred to keep the naming consistent inside it, instead of making it consistent with perf (this is what I was referring to in the cover letter, not just the workqueue API). What do you think?
I also prefer perf's naming conventions, but it'd feel strange to use two different naming conventions inside the same library.
Quoted 11 lines
tools/perf/util/workqueue/threadpool.c
Quoted 8 lines
tools/perf/util/workqueue/threadpool.c new file +7
-0+{ + int ret, t; + struct threadpool_struct *pool = malloc(sizeof(*pool)); + + if (!pool) { + pr_err("threadpool: cannot allocate pool: %s\n", + strerror(errno));oHumm,
pr_err()at this level isn't appropriate, please make callers complain.
ok.
Quoted 9 lines
Quoted 7 lines
+ return NULL;
+ }
+
+ if (n_threads<= 0) {
+ pr_err("threadpool: invalid number of threads: %d\n", + n_threads);
pr_debug()
ok
Quoted 36 lines
Quoted 36 lines
+ goto
out_free_pool;
+ }tools/perf/util/workqueue/threadpool.c new file +33
-0+ + pool->nr_threads = n_threads; + pool->current_task = NULL; + + pool->threads = malloc(n_threads * sizeof(*pool->threads)); + if (!pool->threads) { + pr_err("threadpool: cannot allocate threads: %s\n", + strerror(errno)); + goto out_free_pool; + } + + for (t = 0; t < n_threads; t++) { + pool->threads[t].idx = t; + pool->threads[t].tid = -1; + pool->threads[t].pool = pool; + init_pipes(&pool->threads[t]); + } + + for (t = 0; t < n_threads; t++) { + ret = open_pipes(&pool->threads[t]); + if (ret) + goto out_close_pipes; + } + + pool->status = THREADPOOL_STATUS__STOPPED; + + return pool; + +out_close_pipes: + for (t = 0; t < n_threads; t++) + close_pipes(&pool->threads[t]); + + free(pool->threads);
Quoted 8 lines
Quoted 3 lines
+out_free_pool:
+ free(pool);
+ return NULL;Here we can use ERR_PTR()/PTR_ERR() to let the caller know what was the problem, i.e. we can ditch all the pr_err/pr_debug(), etc and instead have a
threadpool__strerror(struct threadpool *pool, int err)like we have for 'struct evsel', please take a look atevsel__open_strerror().
Thanks, I'll have a look at it.
So, what I sould do is not use pr_* higher than debug inside library code and return meaningful errors through PR_ERR, right?
Quoted 11 lines
tools/perf/util/workqueue/threadpool.c line 152
Quoted 8 lines
tools/perf/util/workqueue/threadpool.c new file +6
-0+} + +/** +* destroy_threadpool - free the @pool and all its resources +*/ +void destroy_threadpool (struct threadpool_struct *pool)void
threadpool__delete(struct threadpool *pool)
Quoted 13 lines
tools/perf/util/workqueue/threadpool.c line 163
Quoted 11 lines
tools/perf/util/workqueue/threadpool.c new file +11
-0+{ + int t; + + if (!pool) + return; + + WARN_ON(pool->status != THREADPOOL_STATUS__STOPPED + && pool->status != THREADPOOL_STATUS__ERROR); + + for (t = 0; t < pool->nr_threads; t++) + close_pipes(&pool->threads[t]);reset pool->threads[t] to -1
already inside close_pipes. I agree it might be confusing without the threadpool_entry__ prefix.
Quoted 5 lines
Quoted 3 lines
+
+ free(pool->threads);zfree
In general, when should I use zfree instead of free?
Quoted 10 lines
tools/perf/util/workqueue/threadpool.c line 172
Quoted 8 lines
+ free(pool);
tools/perf/util/workqueue/threadpool.c new file +6
-0+} + +/** +* threadpool_size - get number of threads in the threadpool +*/ +int threadpool_size (struct threadpool_struct *pool)
threadpool__size()
ok
Thanks,
Riccardo
Quoted 34 lines
Quoted 33 lines
tools/perf/util/workqueue/threadpool.c new file +3
-0+{ + return pool->nr_threads; +}tools/perf/util/workqueue/threadpool.h new file +19
-0@@ -0,0 +1,19 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __WORKQUEUE_THREADPOOL_H +#define __WORKQUEUE_THREADPOOL_H + +struct threadpool_struct ; +struct task_struct ; + +typedef void (*task_func_t )(int tidx,struct task_struct *task); + +struct task_struct { + task_func_t fn; +}; + +extern struct threadpool_struct *create_threadpool (int n_threads); +extern void destroy_threadpool (struct threadpool_struct *pool); + +extern int threadpool_size (struct threadpool_struct *pool); + +#endif /* __WORKQUEUE_THREADPOOL_H */ Signature · 1 line
2.31.1
Re: [RFC PATCH 02/10] perf tests: add test for workqueue
In reply to Arnaldo Carvalho de MeloHi Arnaldo,
Quoted 8 lines · Arnaldo Carvalho de Melo
Em Tue, Jul 13, 2021 at 02:11:13PM +0200, Riccardo Mancini escreveu:
Quoted 7 lines
It will have subtests testing threadpool and workqueue separately. This patch only introduces the first subtest, checking that the threadpool is correctly created and destructed.
This test will be expanded when new functions are added in next patches.Signed-off-by Riccardo Mancini <[email protected]>
<SNIP>
Quoted 11 lines
tools/perf/tests/workqueue.c line 35
Quoted 9 lines
tools/perf/tests/workqueue.c new file +9
-0+ + +static int __test__threadpool (void *_args) +{ + struct threadpool_test_args_t *args = _args; + struct threadpool_struct *pool; + int ret; + + ret = __threadpool__prepare(&pool, args->pool_size);Turn the last three lines into one;
ok
Quoted 15 lines
int ret =
__threadpool__prepare(&pool, args->pool_size);tools/perf/tests/workqueue.c line 43
Quoted 8 lines
+ if (ret)
+ return ret;tools/perf/tests/workqueue.c new file +6
-0+ + ret = __threadpool__teardown(pool); + if (ret) + return ret; + + return 0;Humm, will you add something here in the following csets? Otherwise turn these 5 lines into one:
Quoted 1 line
return
__threadpool__teardown(pool);
ok, it was just copy-paste from above.
Thanks,
Riccardo
Quoted 75 lines
Quoted 74 lines
tools/perf/tests/workqueue.c new file +70
-0+} + +static const struct threadpool_test_args_t threadpool_test_args[] = { + { + .pool_size = 1 + }, + { + .pool_size = 2 + }, + { + .pool_size = 4 + }, + { + .pool_size = 8 + }, + { + .pool_size = 16 + } +}; + +struct test_case { + const char *desc; + int (*func)(void *args); + void *args; + int n_args; + int arg_size; +}; + +static struct test_case workqueue_testcase_table[] = { + { + .desc = "Threadpool", + .func = __test__threadpool, + .args = (void *) threadpool_test_args, + .n_args = (int)ARRAY_SIZE(threadpool_test_args), + .arg_size = sizeof(struct threadpool_test_args_t) + } +}; + + +int test__workqueue (struct test *test__maybe_unused ,int i) +{ + int j, ret = 0; + struct test_case *tc; + + if (i < 0 || i >= (int)ARRAY_SIZE(workqueue_testcase_table)) + return -1; + + tc = &workqueue_testcase_table[i]; + + for (j = 0; j < tc->n_args; j++) { + ret = tc->func(tc->args + (j*tc->arg_size)); + if (ret) + return ret; + } + + return 0; +} + + +int test__workqueue_subtest_get_nr (void ) +{ + return (int)ARRAY_SIZE(workqueue_testcase_table); +} + +const char *test__workqueue_subtest_get_desc (int i) +{ + if (i < 0 || i >= (int)ARRAY_SIZE(workqueue_testcase_table)) + return NULL; + return workqueue_testcase_table[i].desc; +}Signature · 1 line
2.31.1
Re: [RFC PATCH 03/10] perf workqueue: add threadpool start and stop functions
In reply to Arnaldo Carvalho de MeloHi Arnaldo,
Quoted 11 lines · Arnaldo Carvalho de Melo
Em Tue, Jul 13, 2021 at 02:11:14PM +0200, Riccardo Mancini escreveu:
Quoted 10 lines
This patch adds the start and stop functions, alongside the thread function.
Each thread will run until a stop signal is received. Furthermore, start and stop are added to the test.Thread management is based on the prototype from Alexey:
https://lore.kernel.org/lkml/[email protected]/Suggested-by Alexey Bayduraev <[email protected]>Signed-off-by Riccardo Mancini <[email protected]>
<SNIP>
Quoted 8 lines
Quoted 6 lines
static int__threadpool__teardown(struct threadpool_struct *pool){
+ int ret;
+
+ ret =stop_threadpool(pool);int ret =
stop_threadpool(pool);
ok
Quoted 25 lines
Quoted 25 lines
+ TEST_ASSERT_VAL("threadpool start failure", ret == 0); + TEST_ASSERT_VAL("stopped threadpool is ready", + !threadpool_is_ready(pool)); +
destroy_threadpool(pool);
return 0;tools/perf/util/workqueue/threadpool.c +3
-0@@ -4,12 +4,23 @@ #include <unistd.h> #include <errno.h> #include <string.h> +#include <pthread.h> +#include <signal.h> +#include <syscall.h> #include "debug.h" #include "asm/bug.h" #include "threadpool.h"
+#ifndefHAVE_GETTID
Quoted 8 lines
tools/perf/util/workqueue/threadpool.c line 19
Quoted 5 lines
tools/perf/util/workqueue/threadpool.c +5
-0+static inline pid_t gettid (void ) +{ + return (pid_t)syscall(__NR_gettid); +} +#endif Isn't this defined elsewhere? Yeah, when we decide to move it to
tools/lib/workqueue/we'll need it, but for now, reduce patch size.
No, it's just statically defined in tools/perf/jvmti/jvmti_agent.c. I saw there is a libc_compat.h header in tools/include/tools, I could put this definition there, and remove the one from jvmti_agent.c.
<SNIP>
Quoted 5 lines
Quoted 5 lines
+/**
+ *wait_thread- receive ack from thread
+ *
+ * NB: call only from main thread!
+ */
Quoted 8 lines
tools/perf/util/workqueue/threadpool.c static void close_pipes(struct thread_struct *thread) · line 128
Quoted 6 lines
tools/perf/util/workqueue/threadpool.c +6
-0+static int wait_thread (struct thread_struct *thread) +{ + int res; + enum thread_msg msg = THREAD_MSG__UNDEFINED; + + res = read(thread->pipes.from[0], &msg, sizeof(msg));int res =
read(thread->pipes.from[0], &msg, sizeof(msg));
ok
<SNIP>
Quoted 12 lines
tools/perf/util/workqueue/threadpool.c static void close_pipes(struct thread_struct *thread) · line 175
Quoted 9 lines
tools/perf/util/workqueue/threadpool.c +9
-0+/** +* threadpool_thread - function running on thread +* +* This function waits for a signal from main thread to start executing +* a task. +* On completion, it will go back to sleep, waiting for another signal. +* Signals are delivered through pipes. +*/ +static void *threadpool_thread (void *args)
threadpool_function()
Quoted 1 line
ETOMANY 'thread' in a name.
Agreed :)
<SNIP>
Quoted 8 lines
tools/perf/util/workqueue/threadpool.c int threadpool_size(struct threadpool_struct *pool) · line 368
Quoted 6 lines
tools/perf/util/workqueue/threadpool.c +6
-0+/** +* start_threadpool - start all threads in the pool. +* +* The function blocks until all threads are up and running. +*/ +int start_threadpool (struct threadpool_struct *pool)int
threadpool__start(struct threadpool *pool)
ok
Quoted 15 lines
Quoted 15 lines
tools/perf/util/workqueue/threadpool.c +10
-0+{ + int err; + + if (pool->status != THREADPOOL_STATUS__STOPPED) { + pr_err("threadpool: starting not stopped pool\n"); + return -1; + } + + err = __start_threadpool(pool); + pool->status = err ? THREADPOOL_STATUS__ERROR :
THREADPOOL_STATUS__READY;
+ return err;tools/perf/util/workqueue/threadpool.c +2
-0+} +
Quoted 8 lines
tools/perf/util/workqueue/threadpool.c int threadpool_size(struct threadpool_struct *pool) · line 387
Quoted 6 lines
tools/perf/util/workqueue/threadpool.c +6
-0+/** +* stop_threadpool - stop all threads in the pool. +* +* This function blocks waiting for ack from all threads. +*/ +int stop_threadpool (struct threadpool_struct *pool)int
threadpool__stop(struct threadpool *pool)
ok
Quoted 19 lines
Quoted 19 lines
tools/perf/util/workqueue/threadpool.c +15
-0+{ + int t, ret, err = 0; + + if (pool->status != THREADPOOL_STATUS__READY) { + pr_err("threadpool: stopping not ready pool\n"); + return -1; + } + + for (t = 0; t < pool->nr_threads; t++) { + ret = terminate_thread(&pool->threads[t]); + if (ret && !err) + err = -1; + } + + pool->status = err ? THREADPOOL_STATUS__ERROR :
THREADPOOL_STATUS__STOPPED;
+
+ return err;
Quoted 8 lines
tools/perf/util/workqueue/threadpool.c int threadpool_size(struct threadpool_struct *pool) · line 410
Quoted 6 lines
tools/perf/util/workqueue/threadpool.c +6
-0+} + +/** +* threadpool_is_ready - check if the threads are running +*/ +bool threadpool_is_ready (struct threadpool_struct *pool)bool
threadpool__is_ready(struct threadpool *pool)
ok
Thanks,
Riccardo
Quoted 25 lines
Quoted 24 lines
tools/perf/util/workqueue/threadpool.c +3
-0+{ + return pool->status == THREADPOOL_STATUS__READY; +}tools/perf/util/workqueue/threadpool.h +0
-0@@ -14,6 +14,11 @@ struct task_struct { extern struct threadpool_struct *create_threadpool(int n_threads); extern void destroy_threadpool(struct threadpool_struct *pool);tools/perf/util/workqueue/threadpool.h +3
-0+extern int start_threadpool (struct threadpool_struct *pool); +extern int stop_threadpool (struct threadpool_struct *pool); +extern int
threadpool_size(struct threadpool_struct *pool);tools/perf/util/workqueue/threadpool.h +2
-0+extern bool threadpool_is_ready (struct threadpool_struct *pool); +#endif /*
__WORKQUEUE_THREADPOOL_H*/Signature · 1 line
2.31.1
Re: [RFC PATCH 06/10] perf workqueue: introduce workqueue struct
In reply to Arnaldo Carvalho de MeloHi Arnaldo,
thanks again for having a look at this patchset!
Quoted 10 lines · Arnaldo Carvalho de Melo
Em Tue, Jul 13, 2021 at 02:11:17PM +0200, Riccardo Mancini escreveu:
Quoted 9 lines
This patch adds the workqueue definition, along with simple creation and destruction functions.
Furthermore, a simple subtest is added.A workqueue is attached to a pool, on which it executes its workers. Next patches will introduce workers.
Signed-off-by Riccardo Mancini <[email protected]>
<SNIP>
Quoted 11 lines
tools/perf/util/workqueue/workqueue.c line 89
Quoted 7 lines
tools/perf/util/workqueue/workqueue.c new file +7
-0+ +/** +* create_workqueue - create a workqueue associated to @pool +* +* Only one workqueue can execute on a pool at a time. +*/ +struct workqueue_struct *create_workqueue (struct threadpool_struct *pool)I wonder if we should use the exact same kernel signature and not pass a threadpool, essentially having just one threadpool in
tools/perf/that is used bycreate_workqueue(void)?
I wondered the same thing, but I thought that we'd need it to be dynamically created to prevent spawning threads at the beginning that might not even be used.
I think this could be a follow-up patch.
Thanks,
Riccardo
Quoted 133 lines
Quoted 132 lines
+{
+ int err;
+ structworkqueue_struct*wq =malloc(sizeof(struct workqueue_struct));tools/perf/util/workqueue/workqueue.c new file +92
-0+ + + err = pthread_mutex_init(&wq->lock, NULL); + if (err) + goto out_free_wq; + + err = pthread_cond_init(&wq->idle_cond, NULL); + if (err) + goto out_destroy_mutex; + + wq->pool = NULL; + INIT_LIST_HEAD(&wq->busy_list); + INIT_LIST_HEAD(&wq->idle_list); + + INIT_LIST_HEAD(&wq->pending); + + err = pipe(wq->msg_pipe); + if (err) + goto out_destroy_cond; + + wq->task.fn = worker_thread; + + err = attach_threadpool_to_workqueue(wq, pool); + if (err) + goto out_destroy_cond; + + wq->status = WORKQUEUE_STATUS__READY; + + return wq; + +out_destroy_cond: + pthread_cond_destroy(&wq->idle_cond); +out_destroy_mutex: + pthread_mutex_destroy(&wq->lock); +out_free_wq: + free(wq); + return NULL; +} + +/** +* destroy_workqueue - stop @wq workers and destroy @wq +*/ +int destroy_workqueue (struct workqueue_struct *wq) +{ + int err = 0, ret; + + ret = detach_threadpool_from_workqueue(wq); + if (ret) { + pr_err("workqueue: error detaching from threadpool.\n"); + err = -1; + } + + ret = pthread_mutex_destroy(&wq->lock); + if (ret) { + err = -1; + pr_err("workqueue: error pthread_mutex_destroy: %s\n", + strerror(errno)); + } + + ret = pthread_cond_destroy(&wq->idle_cond); + if (ret) { + err = -1; + pr_err("workqueue: error pthread_cond_destroy: %s\n", + strerror(errno)); + } + + ret = close(wq->msg_pipe[0]); + if (ret) { + err = -1; + pr_err("workqueue: error close msg_pipe[0]: %s\n", + strerror(errno)); + } + + ret = close(wq->msg_pipe[1]); + if (ret) { + err = -1; + pr_err("workqueue: error close msg_pipe[1]: %s\n", + strerror(errno)); + } + + free(wq); + + return err; +} + +/** +* workqueue_nr_threads - get size of threadpool underlying @wq +*/ +int workqueue_nr_threads (struct workqueue_struct *wq) +{ + return threadpool_size(wq->pool); +}tools/perf/util/workqueue/workqueue.h new file +20
-0@@ -0,0 +1,24 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __WORKQUEUE_WORKQUEUE_H +#define __WORKQUEUE_WORKQUEUE_H + +#include <stdlib.h> +#include <sys/types.h> +#include <linux/list.h> +#include "threadpool.h" + +struct work_struct ; +typedef void (*work_func_t )(struct work_struct *work); + +struct work_struct { + struct list_head entry; + work_func_t func; +}; + +struct workqueue_struct ; + +extern struct workqueue_struct *create_workqueue(struct threadpool_struct*pool);
tools/perf/util/workqueue/workqueue.h new file +4
-0+extern int destroy_workqueue (struct workqueue_struct *wq); + +extern int workqueue_nr_threads (struct workqueue_struct *wq); +#endif /* __WORKQUEUE_WORKQUEUE_H */ Signature · 1 line
2.31.1
Arnaldo Carvalho de Melo
maintainerRe: [RFC PATCH 03/10] perf workqueue: add threadpool start and stop functions
In reply to Riccardo ManciniEm Thu, Jul 15, 2021 at 06:42:16PM +0200, Riccardo Mancini escreveu:
Quoted 26 lines
Hi Arnaldo,
Quoted 14 lines · Arnaldo Carvalho de Melo
Em Tue, Jul 13, 2021 at 02:11:14PM +0200, Riccardo Mancini escreveu:
Quoted 13 lines
+++
b/tools/perf/util/workqueue/threadpool.ctools/perf/util/workqueue/threadpool.c +3
-0@@ -4,12 +4,23 @@ #include <unistd.h> #include <errno.h> #include <string.h> +#include <pthread.h> +#include <signal.h> +#include <syscall.h> #include "debug.h" #include "asm/bug.h" #include "threadpool.h"
+#ifndefHAVE_GETTIDQuoted 8 lines
tools/perf/util/workqueue/threadpool.c line 19
Quoted 5 lines
tools/perf/util/workqueue/threadpool.c +5
-0+static inline pid_t gettid (void ) +{ + return (pid_t)syscall(__NR_gettid); +} +#endif Isn't this defined elsewhere? Yeah, when we decide to move it to
tools/lib/workqueue/we'll need it, but for now, reduce patch size.
Quoted 3 lines
No, it's just statically defined in
tools/perf/jvmti/jvmti_agent.c. I saw there is alibc_compat.hheader intools/include/tools, I could put this definition there, and remove the one fromjvmti_agent.c.
Please, do it as a prep patch.
Thanks,
- Arnaldo
Arnaldo Carvalho de Melo
maintainerRe: [RFC PATCH 06/10] perf workqueue: introduce workqueue struct
In reply to Riccardo ManciniEm Thu, Jul 15, 2021 at 06:49:57PM +0200, Riccardo Mancini escreveu:
Quoted 27 lines
Hi Arnaldo,
thanks again for having a look at this patchset!Quoted 10 lines · Arnaldo Carvalho de Melo
Em Tue, Jul 13, 2021 at 02:11:17PM +0200, Riccardo Mancini escreveu:
Quoted 9 lines
This patch adds the workqueue definition, along with simple creation and destruction functions.
Furthermore, a simple subtest is added.A workqueue is attached to a pool, on which it executes its workers. Next patches will introduce workers.
Signed-off-by Riccardo Mancini <[email protected]><SNIP>
Quoted 11 lines
tools/perf/util/workqueue/workqueue.c line 89
Quoted 7 lines
tools/perf/util/workqueue/workqueue.c new file +7
-0+ +/** +* create_workqueue - create a workqueue associated to @pool +* +* Only one workqueue can execute on a pool at a time. +*/ +struct workqueue_struct *create_workqueue (struct threadpool_struct *pool)I wonder if we should use the exact same kernel signature and not pass a threadpool, essentially having just one threadpool in
tools/perf/that is used bycreate_workqueue(void)?
Quoted 4 lines
I wondered the same thing, but I thought that we'd need it to be dynamically created to prevent spawning threads at the beginning that might not even be used.
I think this could be a follow-up patch.
I see your point.
My practice is to use the perf convention for things that don't come from the kernel, and if we do as I suggested, tooling will probably not even see this threadpool struct, i.e. just the workqueue APIs are exposed and workqueue_create() will notice that a threadpool is not yet created, doing its creation behind tooling's back.
- Arnaldo
Quoted 138 lines
Thanks,
RiccardoQuoted 133 lines
Quoted 132 lines
+{
+ int err;
+ structworkqueue_struct*wq =malloc(sizeof(struct workqueue_struct));tools/perf/util/workqueue/workqueue.c new file +92
-0+ + + err = pthread_mutex_init(&wq->lock, NULL); + if (err) + goto out_free_wq; + + err = pthread_cond_init(&wq->idle_cond, NULL); + if (err) + goto out_destroy_mutex; + + wq->pool = NULL; + INIT_LIST_HEAD(&wq->busy_list); + INIT_LIST_HEAD(&wq->idle_list); + + INIT_LIST_HEAD(&wq->pending); + + err = pipe(wq->msg_pipe); + if (err) + goto out_destroy_cond; + + wq->task.fn = worker_thread; + + err = attach_threadpool_to_workqueue(wq, pool); + if (err) + goto out_destroy_cond; + + wq->status = WORKQUEUE_STATUS__READY; + + return wq; + +out_destroy_cond: + pthread_cond_destroy(&wq->idle_cond); +out_destroy_mutex: + pthread_mutex_destroy(&wq->lock); +out_free_wq: + free(wq); + return NULL; +} + +/** +* destroy_workqueue - stop @wq workers and destroy @wq +*/ +int destroy_workqueue (struct workqueue_struct *wq) +{ + int err = 0, ret; + + ret = detach_threadpool_from_workqueue(wq); + if (ret) { + pr_err("workqueue: error detaching from threadpool.\n"); + err = -1; + } + + ret = pthread_mutex_destroy(&wq->lock); + if (ret) { + err = -1; + pr_err("workqueue: error pthread_mutex_destroy: %s\n", + strerror(errno)); + } + + ret = pthread_cond_destroy(&wq->idle_cond); + if (ret) { + err = -1; + pr_err("workqueue: error pthread_cond_destroy: %s\n", + strerror(errno)); + } + + ret = close(wq->msg_pipe[0]); + if (ret) { + err = -1; + pr_err("workqueue: error close msg_pipe[0]: %s\n", + strerror(errno)); + } + + ret = close(wq->msg_pipe[1]); + if (ret) { + err = -1; + pr_err("workqueue: error close msg_pipe[1]: %s\n", + strerror(errno)); + } + + free(wq); + + return err; +} + +/** +* workqueue_nr_threads - get size of threadpool underlying @wq +*/ +int workqueue_nr_threads (struct workqueue_struct *wq) +{ + return threadpool_size(wq->pool); +}tools/perf/util/workqueue/workqueue.h new file +20
-0@@ -0,0 +1,24 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __WORKQUEUE_WORKQUEUE_H +#define __WORKQUEUE_WORKQUEUE_H + +#include <stdlib.h> +#include <sys/types.h> +#include <linux/list.h> +#include "threadpool.h" + +struct work_struct ; +typedef void (*work_func_t )(struct work_struct *work); + +struct work_struct { + struct list_head entry; + work_func_t func; +}; + +struct workqueue_struct ; + +extern struct workqueue_struct *create_workqueue(struct threadpool_struct*pool);
tools/perf/util/workqueue/workqueue.h new file +4
-0+extern int destroy_workqueue (struct workqueue_struct *wq); + +extern int workqueue_nr_threads (struct workqueue_struct *wq); +#endif /* __WORKQUEUE_WORKQUEUE_H */ Signature · 1 line
2.31.1
Signature · 1 line
- Arnaldo
Arnaldo Carvalho de Melo
maintainerRe: [RFC PATCH 01/10] perf workqueue: threadpool creation and destruction
In reply to Riccardo ManciniEm Thu, Jul 15, 2021 at 06:31:07PM +0200, Riccardo Mancini escreveu:
Quoted 97 lines
Hi Arnaldo,
thanks for reviewing the patch!
On Wed, 2021-07-14 at 11:16 -0300, Arnaldo Carvalho de Melo wrote:
<SNIP>Quoted 10 lines
tools/perf/util/workqueue/threadpool.c line 17
Quoted 8 lines
tools/perf/util/workqueue/threadpool.c new file +8
-0+ +enum threadpool_status { + THREADPOOL_STATUS__STOPPED, /* no threads */ + THREADPOOL_STATUS__ERROR, /* errors */ + THREADPOOL_STATUS__MAX +}; + +struct threadpool_struct {Can this be just 'struct threadpool'? I think its descriptive enough:
I agree, but I wanted to keep the naming consistent between
workqueue.candthreadpool.c.Quoted 27 lines
Quoted 16 lines
+ int nr_threads; /* number of threads in the pool */
+ struct thread_struct *threads; /* array of threads in the pool */
+ struct task_struct *current_task; /* current executing function
*/
+ enum threadpool_status status; /* current status of the pool
*/tools/perf/util/workqueue/threadpool.c new file +4
-0+}; + +struct thread_struct { + int idx; /* idx of thread in pool-Quoted 1 line
threads */
Quoted 10 lines
+ pid_t tid; /* tid of thread */ + struct threadpool_struct *pool; /* parent threadpool */ + struct {
+ int from[2]; /* messages from thread (acks)
*/
+ int to[2]; /* messages to thread (commands) */
+ } pipes;
+};Quoted 3 lines
This one, since we have already a 'struct thread' in
tools/perf, to represent aPERF_RECORD_FORK, perhaps we can call it 'structthreadpool_entry'?Agreed.
Quoted 22 lines
Quoted 7 lines
tools/perf/util/workqueue/threadpool.c new file +6
-0+ +/** +* init_pipes - initialize all pipes of @thread +*/ +static void init_pipes (struct thread_struct *thread) +{tools/perf/util/workqueue/threadpool.c line 48
Quoted 10 lines
tools/perf/util/workqueue/threadpool.c new file +10
-0+ thread->pipes.from[0] = -1; + thread->pipes.from[1] = -1; + thread->pipes.to[0] = -1; + thread->pipes.to[1] = -1; +} + +/** +* open_pipes - open all pipes of @thread +*/ +static int open_pipes (struct thread_struct *thread)Here please:
threadpool_entry__open_pipes()Quoted 7 lines
Its longer, but helps with ctags/cscope navigation and we can go directly to it via:
:ta threadpool_entry__open_p<TAB>
While 'ta:
open_pipes' may bo to various places where this idiom is used.Agreed.
<SNIP>
Quoted 10 lines
tools/perf/util/workqueue/threadpool.c line 95
Quoted 4 lines
tools/perf/util/workqueue/threadpool.c new file +4
-0+/** +* create_threadpool - create a fixed threadpool with @n_threads threads +*/ +struct threadpool_struct *create_threadpool (int n_threads)Is this already something the kernel has and thus we should keep the naming? I couldn't find it in the kernel, so please name it:
struct threadpool *threadpool__new(int nthreads)
Quoted 7 lines
As before, I did this to keep consistency with workqueue.
Since this threadpool+workqueue can be a standalone library, I preferred to keep the naming consistent inside it, instead of making it consistent with perf (this is what I was referring to in the cover letter, not just the workqueue API). What do you think?
I also prefer perf's naming conventions, but it'd feel strange to use two different naming conventions inside the same library.
See my comment on the other message about this naming dilemma :-)
Quoted 71 lines
Quoted 11 lines
tools/perf/util/workqueue/threadpool.c
Quoted 8 lines
tools/perf/util/workqueue/threadpool.c new file +7
-0+{ + int ret, t; + struct threadpool_struct *pool = malloc(sizeof(*pool)); + + if (!pool) { + pr_err("threadpool: cannot allocate pool: %s\n", + strerror(errno));oHumm,
pr_err()at this level isn't appropriate, please make callers complain.ok.
Quoted 9 lines
Quoted 7 lines
+ return NULL;
+ }
+
+ if (n_threads<= 0) {
+ pr_err("threadpool: invalid number of threads: %d\n", + n_threads);
pr_debug()ok
Quoted 36 lines
Quoted 36 lines
+ goto
out_free_pool;
+ }tools/perf/util/workqueue/threadpool.c new file +33
-0+ + pool->nr_threads = n_threads; + pool->current_task = NULL; + + pool->threads = malloc(n_threads * sizeof(*pool->threads)); + if (!pool->threads) { + pr_err("threadpool: cannot allocate threads: %s\n", + strerror(errno)); + goto out_free_pool; + } + + for (t = 0; t < n_threads; t++) { + pool->threads[t].idx = t; + pool->threads[t].tid = -1; + pool->threads[t].pool = pool; + init_pipes(&pool->threads[t]); + } + + for (t = 0; t < n_threads; t++) { + ret = open_pipes(&pool->threads[t]); + if (ret) + goto out_close_pipes; + } + + pool->status = THREADPOOL_STATUS__STOPPED; + + return pool; + +out_close_pipes: + for (t = 0; t < n_threads; t++) + close_pipes(&pool->threads[t]); + + free(pool->threads);Quoted 8 lines
Quoted 3 lines
+out_free_pool:
+ free(pool);
+ return NULL;Here we can use ERR_PTR()/PTR_ERR() to let the caller know what was the problem, i.e. we can ditch all the pr_err/pr_debug(), etc and instead have a
threadpool__strerror(struct threadpool *pool, int err)like we have for 'struct evsel', please take a look atevsel__open_strerror().
Quoted 3 lines
Thanks, I'll have a look at it.
So, what I sould do is not use pr_* higher than debug inside library code and return meaningful errors throughPR_ERR, right?
Right.
Quoted 89 lines
Quoted 11 lines
tools/perf/util/workqueue/threadpool.c line 152
Quoted 8 lines
tools/perf/util/workqueue/threadpool.c new file +6
-0+} + +/** +* destroy_threadpool - free the @pool and all its resources +*/ +void destroy_threadpool (struct threadpool_struct *pool)void
threadpool__delete(struct threadpool *pool)Quoted 13 lines
tools/perf/util/workqueue/threadpool.c line 163
Quoted 11 lines
tools/perf/util/workqueue/threadpool.c new file +11
-0+{ + int t; + + if (!pool) + return; + + WARN_ON(pool->status != THREADPOOL_STATUS__STOPPED + && pool->status != THREADPOOL_STATUS__ERROR); + + for (t = 0; t < pool->nr_threads; t++) + close_pipes(&pool->threads[t]);reset pool->threads[t] to -1
already inside
close_pipes. I agree it might be confusing without thethreadpool_entry__prefix.Quoted 5 lines
Quoted 3 lines
+
+ free(pool->threads);zfree
In general, when should I use zfree instead of free?
Quoted 10 lines
tools/perf/util/workqueue/threadpool.c line 172
Quoted 8 lines
+ free(pool);
tools/perf/util/workqueue/threadpool.c new file +6
-0+} + +/** +* threadpool_size - get number of threads in the threadpool +*/ +int threadpool_size (struct threadpool_struct *pool)
threadpool__size()ok
Thanks,
RiccardoQuoted 34 lines
Quoted 33 lines
tools/perf/util/workqueue/threadpool.c new file +3
-0+{ + return pool->nr_threads; +}tools/perf/util/workqueue/threadpool.h new file +19
-0@@ -0,0 +1,19 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __WORKQUEUE_THREADPOOL_H +#define __WORKQUEUE_THREADPOOL_H + +struct threadpool_struct ; +struct task_struct ; + +typedef void (*task_func_t )(int tidx,struct task_struct *task); + +struct task_struct { + task_func_t fn; +}; + +extern struct threadpool_struct *create_threadpool (int n_threads); +extern void destroy_threadpool (struct threadpool_struct *pool); + +extern int threadpool_size (struct threadpool_struct *pool); + +#endif /* __WORKQUEUE_THREADPOOL_H */ Signature · 1 line
2.31.1
Signature · 1 line
- Arnaldo
Namhyung Kim
maintainerRe: [RFC PATCH 01/10] perf workqueue: threadpool creation and destruction
In reply to Arnaldo Carvalho de MeloHi Riccardo and Arnaldo,
Quoted 66 lines · Arnaldo Carvalho de Melo
Em Tue, Jul 13, 2021 at 02:11:12PM +0200, Riccardo Mancini escreveu:
Quoted 64 lines
The workqueue library is made up by two components:
- threadpool: handles the lifetime of the threads
- workqueue: handles work distribution among the threadsThis first patch introduces the threadpool, starting from its creation and destruction functions.
Thread management is based on the prototype from Alexey:
https://lore.kernel.org/lkml/[email protected]/Each thread in the threadpool executes the same function (aka task) with a different argument tidx.
Threads use a pair of pipes to communicate with the main process. The threadpool is static (all threads will be spawned at the same time). Future work could include making it resizable and adding affinity support (as in Alexey prototype).Suggested-by Alexey Bayduraev <[email protected]>Signed-off-by Riccardo Mancini <[email protected]>--- tools/perf/util/Build | 1 + tools/perf/util/workqueue/Build | 1 + tools/perf/util/workqueue/threadpool.c | 175 +++++++++++++++++++++++++ tools/perf/util/workqueue/threadpool.h | 19 +++ 4 files changed, 196 insertions(+) create mode 100644 tools/perf/util/workqueue/Build create mode 100644 tools/perf/util/workqueue/threadpool.c create mode 100644 tools/perf/util/workqueue/threadpool.htools/perf/util/Build +1
-0@@ -180,6 +180,7 @@ perf-$(CONFIG_LIBBABELTRACE) += data-convert-bt.o perf-y += data-convert-json.o perf-y += scripting-engines/ +perf-y += workqueue/ perf-$(CONFIG_ZLIB) += zlib.o perf-$(CONFIG_LZMA) += lzma.otools/perf/util/workqueue/Build new file +1
-0@@ -0,0 +1 @@ +perf-y += threadpool.otools/perf/util/workqueue/threadpool.c new file +11
-0@@ -0,0 +1,175 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <stdlib.h> +#include <stdio.h> +#include <unistd.h> +#include <errno.h> +#include <string.h> +#include "debug.h" +#include "asm/bug.h" +#include "threadpool.h" + +enum threadpool_status {
Quoted 12 lines
tools/perf/util/workqueue/threadpool.c line 17
Quoted 6 lines
tools/perf/util/workqueue/threadpool.c new file +6
-0+THREADPOOL_STATUS__STOPPED ,/* no threads */ +THREADPOOL_STATUS__ERROR ,/* errors */ +THREADPOOL_STATUS__MAX +}; + +struct threadpool_struct {Can this be just 'struct threadpool'? I think its descriptive enough:
Quoted 3 lines
tools/perf/util/workqueue/threadpool.c new file +3
-0+int nr_threads;/* number of threads in the pool */ +struct thread_struct *threads;/* array of threads in the pool */ +struct task_struct *current_task;/* current executing function */
Does this mean it can only have a single function to run? Why do we need it?
Quoted 10 lines
tools/perf/util/workqueue/threadpool.c line 30
Quoted 10 lines
tools/perf/util/workqueue/threadpool.c new file +10
-0+enum threadpool_status status;/* current status of the pool */ +}; + +struct thread_struct { +int idx;/* idx of thread in pool->threads */ +pid_t tid;/* tid of thread */ +struct threadpool_struct *pool;/* parent threadpool */ +struct { +int from[2 ];/* messages from thread (acks) */ +int to[2 ];/* messages to thread (commands) */
It can be confusing if you think from the main thread. Maybe 'ack' and 'cmd' would be better.
Quoted 5 lines
tools/perf/util/workqueue/threadpool.c line 32
Quoted 2 lines
tools/perf/util/workqueue/threadpool.c new file +2
-0+ } pipes; +};This one, since we have already a 'struct thread' in
tools/perf, to represent aPERF_RECORD_FORK, perhaps we can call it 'structthreadpool_entry'?
I think we can even use 'worker' instead of 'thread' but it requires huge renaming and conflicts so I won't insist on it strongly. :)
Thanks,
Namhyung
Namhyung Kim
maintainerRe: [RFC PATCH 03/10] perf workqueue: add threadpool start and stop functions
In reply to Riccardo ManciniQuoted 35 lines · Riccardo Mancini
This patch adds the start and stop functions, alongside the thread function.
Each thread will run until a stop signal is received. Furthermore, start and stop are added to the test.Thread management is based on the prototype from Alexey:
https://lore.kernel.org/lkml/[email protected]/Suggested-by Alexey Bayduraev <[email protected]>Signed-off-by Riccardo Mancini <[email protected]>--- tools/perf/tests/workqueue.c | 13 ++ tools/perf/util/workqueue/threadpool.c | 238 +++++++++++++++++++++++++ tools/perf/util/workqueue/threadpool.h | 5 + 3 files changed, 256 insertions(+)tools/perf/tests/workqueue.c +5
-0@@ -10,16 +10,29 @@ struct threadpool_test_args_t {static int __threadpool__prepare (struct threadpool_struct **pool,int pool_size) { +int ret; + *pool =create_threadpool (pool_size);TEST_ASSERT_VAL ("threadpool creation failure" , *pool !=NULL );TEST_ASSERT_VAL ("threadpool size is wrong" ,threadpool_size (*pool) == pool_size); + ret =start_threadpool (*pool); +TEST_ASSERT_VAL ("threadpool start failure" , ret ==0 ); +TEST_ASSERT_VAL ("threadpool is not ready" ,threadpool_is_ready (*pool));
tools/perf/tests/workqueue.c struct threadpool_test_args_t { · line 32
Quoted 10 lines
tools/perf/tests/workqueue.c +5
-0+return 0 ; }static int __threadpool__teardown (struct threadpool_struct *pool) { +int ret; + + ret =stop_threadpool (pool); +TEST_ASSERT_VAL ("threadpool start failure" , ret ==0 );
s/start/stop/
Quoted 10 lines
tools/perf/tests/workqueue.c +3
-0+TEST_ASSERT_VAL ("stopped threadpool is ready" , + !threadpool_is_ready (pool)); +destroy_threadpool (pool);return 0 ;tools/perf/util/workqueue/threadpool.c +0
-0
[SNIP]
tools/perf/util/workqueue/threadpool.c static void close_pipes(struct thread_struct *thread) · line 129
Quoted 12 lines
tools/perf/util/workqueue/threadpool.c +12
-0+/** +* wait_thread - receive ack from thread +* +* NB: call only from main thread! +*/ +static int wait_thread (struct thread_struct *thread) +{ +int res; +enum thread_msg msg =THREAD_MSG__UNDEFINED ; + + res =read (thread->pipes.from[0 ], &msg,sizeof (msg)); +if (res <0 ) {
Maybe it needs to handle -EINTR.
Quoted 133 lines
tools/perf/util/workqueue/threadpool.c +126
-0+pr_err ("threadpool: failed to recv msg from tid=%d: %s\n" , + thread->tid,strerror (errno)); +return -1 ; + } +if (msg !=THREAD_MSG__ACK ) { +pr_err ("threadpool: received unexpected msg from tid=%d: %s\n" , + thread->tid, thread_msg_tags[msg]); +return -1 ; + } + +pr_debug2 ("threadpool: received ack from tid=%d\n" , thread->tid); + +return 0 ; +} + +/** +* terminate_thread - send stop signal to thread and wait for ack +* +* NB: call only from main thread! +*/ +static int terminate_thread (struct thread_struct *thread) +{ +int res; +enum thread_msg msg =THREAD_MSG__STOP ; + + res =write (thread->pipes.to[1 ], &msg,sizeof (msg)); +if (res <0 ) { +pr_err ("threadpool: error sending stop msg to tid=%d: %s\n" , + thread->tid,strerror (errno)); +return res; + } + + res =wait_thread (thread); + +return res; +} + +/** +* threadpool_thread - function running on thread +* +* This function waits for a signal from main thread to start executing +* a task. +* On completion, it will go back to sleep, waiting for another signal. +* Signals are delivered through pipes. +*/ +static void *threadpool_thread (void *args) +{ +struct thread_struct *thread = (struct thread_struct *) args; +enum thread_msg msg; +int err; + + thread->tid =gettid (); + +pr_debug2 ("threadpool[%d]: started\n" , thread->tid); + +for (;;) { + msg =THREAD_MSG__ACK ; + err =write (thread->pipes.from[1 ], &msg,sizeof (msg)); +if (err ==-1 ) { +pr_err ("threadpool[%d]: failed to send ack: %s\n" , + thread->tid,strerror (errno)); +break ; + } + + msg =THREAD_MSG__UNDEFINED ; + err =read (thread->pipes.to[0 ], &msg,sizeof (msg)); +if (err <0 ) { +pr_err ("threadpool[%d]: error receiving msg: %s\n" , + thread->tid,strerror (errno)); +break ; + } + +if (msg !=THREAD_MSG__WAKE && msg !=THREAD_MSG__STOP ) { +pr_err ("threadpool[%d]: received unexpected msg: %s\n" , + thread->tid, thread_msg_tags[msg]); +break ; + } + +if (msg ==THREAD_MSG__STOP ) +break ; + } + +pr_debug2 ("threadpool[%d]: exit\n" , thread->tid); + + msg =THREAD_MSG__ACK ; + err =write (thread->pipes.from[1 ], &msg,sizeof (msg)); +if (err ==-1 ) { +pr_err ("threadpool[%d]: failed to send ack: %s\n" , + thread->tid,strerror (errno)); +return NULL ; + } + +return NULL ; +} +/** * create_threadpool - create a fixed threadpool with @n_threads threads */ @@ -173,3 +306,108 @@ int threadpool_size(struct threadpool_struct *pool) {return pool->nr_threads; } + +/** +* __start_threadpool - start all threads in the pool. +* +* This function does not change @pool->status. +*/ +static int __start_threadpool (struct threadpool_struct *pool) +{ +int t, tt, ret =0 , nr_threads = pool->nr_threads; +sigset_t full, mask; +pthread_t handle; +pthread_attr_t attrs; + +sigfillset (&full); +if (sigprocmask (SIG_SETMASK , &full, &mask)) { +pr_err ("Failed to block signals on threads start: %s\n" , +strerror (errno)); +return -1 ; + } + +pthread_attr_init (&attrs); +pthread_attr_setdetachstate (&attrs,PTHREAD_CREATE_DETACHED ); + +for (t =0 ; t < nr_threads; t++) { +struct thread_struct *thread = &pool->threads[t]; + +if (pthread_create (&handle, &attrs, threadpool_thread, thread)) { +for (tt =1 ; tt < t; tt++) +terminate_thread (thread); +pr_err ("Failed to start threads: %s\n" ,strerror (errno)); + ret =-1 ;
tools/perf/util/workqueue/threadpool.c int threadpool_size(struct threadpool_struct *pool) · line 349
Quoted 10 lines
tools/perf/util/workqueue/threadpool.c +10
-0+ goto out_free_attr; + } + +if (wait_thread (thread)) { +for (tt =1 ; tt <= t; tt++) +terminate_thread (thread); + ret =-1 ; + goto out_free_attr; + } + }
Isn't it better doing this way?
for (t = 0; t < nr_threads; t++) {
pthread_create(t)
}
for (t = 0; t < nr_threads; t++) {
wait_thread(t)
}
Thanks,
Namhyung
Quoted 13 lines
tools/perf/util/workqueue/threadpool.c +13
-0+ +out_free_attr: +pthread_attr_destroy (&attrs); + +if (sigprocmask (SIG_SETMASK , &mask,NULL )) { +pr_err ("Failed to unblock signals on threads start: %s\n" , +strerror (errno)); + ret =-1 ; + } + +return ret; +} +
Namhyung Kim
maintainerRe: [RFC PATCH 04/10] perf workqueue: add threadpool execute and wait functions
In reply to Riccardo ManciniQuoted 40 lines · Riccardo Mancini
This patch adds:
-execute_in_threadpool: assigns a task to the threads to execute asynchronously.
-wait_threadpool: waits for the task to complete on all threads.
Furthermore, testing for these new functions is added.This patch completes the threadpool.
Signed-off-by Riccardo Mancini <[email protected]>--- tools/perf/tests/workqueue.c | 86 ++++++++++++++++++++- tools/perf/util/workqueue/threadpool.c | 103 +++++++++++++++++++++++++ tools/perf/util/workqueue/threadpool.h | 5 ++ 3 files changed, 193 insertions(+), 1 deletion(-)tools/perf/tests/workqueue.c +9
-0@@ -1,13 +1,59 @@// SPDX-License-Identifier: GPL-2.0 +#include <stdlib.h> #include <linux/kernel.h> +#include <linux/zalloc.h> #include "tests.h" #include "util/debug.h" #include "util/workqueue/threadpool.h" +#define DUMMY_FACTOR 100000 +#define N_DUMMY_WORK_SIZES 7 +struct threadpool_test_args_t {int pool_size; }; +struct test_task { +struct task_struct task; +int n_threads; +int *array;
tools/perf/tests/workqueue.c line 29
Quoted 10 lines
tools/perf/tests/workqueue.c +10
-0+}; + +/** +* dummy_work - calculates DUMMY_FACTOR * (idx % N_DUMMY_WORK_SIZES) inefficiently +* +* This function uses modulus to create work items of different sizes. +*/ +static void dummy_work (int idx) +{ +int prod =0 ;
I'm not sure but having 'volatile' would prevent some kind of possible compiler optimizations..
Quoted 51 lines
tools/perf/tests/workqueue.c +42
-0+int k = idx %N_DUMMY_WORK_SIZES ; +int i, j; + +for (i =0 ; i <DUMMY_FACTOR ; i++) +for (j =0 ; j < k; j++) + prod ++; + +pr_debug3 ("dummy: %d * %d = %d\n" ,DUMMY_FACTOR , k, prod); +} + +static void test_task_fn1 (int tidx,struct task_struct *task) +{ + struct test_task *mtask = container_of(task, struct test_task, task); + +dummy_work (tidx); + mtask->array[tidx] = tidx+1 ; +} + +static void test_task_fn2 (int tidx,struct task_struct *task) +{ + struct test_task *mtask = container_of(task, struct test_task, task); + +dummy_work (tidx); + mtask->array[tidx] = tidx*2 ; +} + +static int __threadpool__prepare (struct threadpool_struct **pool,int pool_size) {int ret;@@ -38,21 +84,59 @@ static int __threadpool__teardown(struct threadpool_struct *pool)return 0 ; } +static int __threadpool__exec_wait (struct threadpool_struct *pool, +struct task_struct *task) +{ +int ret; + + ret =execute_in_threadpool (pool, task); +TEST_ASSERT_VAL ("threadpool execute failure" , ret ==0 ); +TEST_ASSERT_VAL ("threadpool is not executing" ,threadpool_is_busy (pool)); + + ret =wait_threadpool (pool); +TEST_ASSERT_VAL ("threadpool wait failure" , ret ==0 ); +TEST_ASSERT_VAL ("waited threadpool is not ready" ,threadpool_is_ready (pool)); + +return 0 ; +}static int __test__threadpool (void *_args)
tools/perf/tests/workqueue.c static int __threadpool__teardown(struct threadpool_struct *pool) · line 112
Quoted 10 lines
tools/perf/tests/workqueue.c +6
-1{struct threadpool_test_args_t *args = _args;struct threadpool_struct *pool; -int ret; +int ret, i; +struct test_task task; + + task.task.fn = test_task_fn1; + task.n_threads = args->pool_size; + task.array =calloc (args->pool_size,sizeof (*task.array));
Need to check the return value.
Quoted 16 lines
tools/perf/tests/workqueue.c +11
-0ret =__threadpool__prepare (&pool, args->pool_size);if (ret)return ret; + ret =__threadpool__exec_wait (pool, &task.task); +if (ret) +return ret; + +for (i =0 ; i < args->pool_size; i++) +TEST_ASSERT_VAL ("failed array check (1)" , task.array[i] == i+1 ); + + task.task.fn = test_task_fn2; + + ret =__threadpool__exec_wait (pool, &task.task); +if (ret)
tools/perf/tests/workqueue.c static int __threadpool__teardown(struct threadpool_struct *pool) · line 138
Quoted 10 lines
tools/perf/tests/workqueue.c +6
-0+return ret; + +for (i =0 ; i < args->pool_size; i++) +TEST_ASSERT_VAL ("failed array check (2)" , task.array[i] ==2 *i); + ret =__threadpool__teardown (pool);if (ret)return ret; +free (task.array);
All previous returns will leak it.
Thanks,
Namhyung
Re: [RFC PATCH 01/10] perf workqueue: threadpool creation and destruction
In reply to Namhyung KimHi Namhyung,
thanks for the review.
Quoted 33 lines · Namhyung Kim
Hi Riccardo and Arnaldo,
Quoted 29 lines · Arnaldo Carvalho de Melo
Em Tue, Jul 13, 2021 at 02:11:12PM +0200, Riccardo Mancini escreveu:
Quoted 27 lines
The workqueue library is made up by two components:
- threadpool: handles the lifetime of the threads
- workqueue: handles work distribution among the threadsThis first patch introduces the threadpool, starting from its creation and destruction functions.
Thread management is based on the prototype from Alexey:
https://lore.kernel.org/lkml/[email protected]/Each thread in the threadpool executes the same function (aka task) with a different argument tidx.
Threads use a pair of pipes to communicate with the main process. The threadpool is static (all threads will be spawned at the same time). Future work could include making it resizable and adding affinity support (as in Alexey prototype).Suggested-by Alexey Bayduraev <[email protected]>Signed-off-by Riccardo Mancini <[email protected]>---
tools/perf/util/Build | 1 + tools/perf/util/workqueue/Build | 1 +tools/perf/util/workqueue/threadpool.c| 175 +++++++++++++++++++++++++tools/perf/util/workqueue/threadpool.h| 19 +++4 files changed, 196 insertions(+)create mode 100644
tools/perf/util/workqueue/Buildcreate mode 100644tools/perf/util/workqueue/threadpool.ccreate mode 100644tools/perf/util/workqueue/threadpool.h
<SNIP>
Quoted 5 lines
Quoted 5 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+ +struct threadpool_struct {Can this be just 'struct threadpool'? I think its descriptive enough:
Quoted 8 lines
Quoted 6 lines
Quoted 6 lines
+ int nr_threads; /* number of threads in the pool */
+ struct thread_struct *threads; /* array of threads in the pool */
+ struct task_struct *current_task; /* current executing function */Does this mean it can only have a single function to run?
Yes.
Quoted 1 line
Why do we need it?
My idea is to separate the workqueue from the actual implementation of the threads. This way, when the function executing on the threadpool ends, the threads are kept alive to execute new work.
By adding this additional layer of abstraction, we can achieve more flexibility. For example, the use-case I have in mind is to recycle the same threadpool for both Alexey's threaded trace and the workqueue.
I don't think this could be easily achieved with just the workqueue since the perf-record threads are not just a task that needs to be executed by they have specific affinities to be respected.
What are your thoughts?
Quoted 17 lines
Quoted 8 lines
Quoted 8 lines
+ enum threadpool_status status; /* current status of the pool */
tools/perf/util/workqueue/threadpool.c new file +4
-0+}; + +struct thread_struct { + int idx; /* idx of thread in pool-Quoted 8 lines
Quoted 8 lines
Quoted 1 line
threads */
+ pid_t tid; /* tid of thread */ + struct threadpool_struct *pool; /* parent threadpool */ + struct {
+ int from[2]; /* messages from thread (acks) */
+ int to[2]; /* messages to thread (commands) */
Quoted 2 lines
It can be confusing if you think from the main thread. Maybe 'ack' and 'cmd' would be better.
Agreed.
Quoted 11 lines
Quoted 8 lines
Quoted 4 lines
+ } pipes;
+};This one, since we have already a 'struct thread' in
tools/perf, to represent aPERF_RECORD_FORK, perhaps we can call it 'structthreadpool_entry'?I think we can even use 'worker' instead of 'thread' but it requires huge renaming and conflicts so I won't insist on it strongly. :)
Also, worker internally conflicts with the workqueue's worker, which runs on a (threadpool-)thread.
Another name I had in mind is pool_thread to prevent having too many 'thread' in the name, but it might be confusing.
I think threadpool_entry is fine.
I have another question.
In general, when should I use zfree instead of free?
Thanks,
Riccardo
Quoted 3 lines
Thanks,
Namhyung
Re: [RFC PATCH 03/10] perf workqueue: add threadpool start and stop functions
In reply to Namhyung KimHi Namhyung,
Quoted 42 lines · Namhyung Kim
Quoted 41 lines · Riccardo Mancini
This patch adds the start and stop functions, alongside the thread function.
Each thread will run until a stop signal is received. Furthermore, start and stop are added to the test.Thread management is based on the prototype from Alexey:
https://lore.kernel.org/lkml/[email protected]/Suggested-by Alexey Bayduraev <[email protected]>Signed-off-by Riccardo Mancini <[email protected]>---
tools/perf/tests/workqueue.c | 13 ++tools/perf/util/workqueue/threadpool.c| 238 +++++++++++++++++++++++++tools/perf/util/workqueue/threadpool.h| 5 +3 files changed, 256 insertions(+)tools/perf/tests/workqueue.c +0
-0@@ -10,16 +10,29 @@ struct threadpool_test_args_t {static int
__threadpool__prepare(struct threadpool_struct **pool, int pool_size)
{
+ int ret;
+
*pool =create_threadpool(pool_size);TEST_ASSERT_VAL("threadpool creation failure", *pool != NULL);TEST_ASSERT_VAL("threadpool size is wrong", threadpool_size(*pool) == pool_size);+ ret =
start_threadpool(*pool);
+TEST_ASSERT_VAL("threadpool start failure", ret == 0); +TEST_ASSERT_VAL("threadpool is not ready", threadpool_is_ready(*pool));
+
return 0;
}
Quoted 8 lines
Quoted 6 lines
static int
__threadpool__teardown(struct threadpool_struct *pool){
+ int ret;
+
+ ret =stop_threadpool(pool);
+TEST_ASSERT_VAL("threadpool start failure", ret == 0);s/start/stop/
Thanks.
Quoted 13 lines
Quoted 12 lines
+ TEST_ASSERT_VAL("stopped threadpool is ready", + !threadpool_is_ready(pool)); +
destroy_threadpool(pool);return 0;
tools/perf/util/workqueue/threadpool.c +0
-0[SNIP]
Quoted 14 lines
tools/perf/util/workqueue/threadpool.c static void close_pipes(struct thread_struct *thread) · line 129
Quoted 12 lines
tools/perf/util/workqueue/threadpool.c +12
-0+/** +* wait_thread - receive ack from thread +* +* NB: call only from main thread! +*/ +static int wait_thread (struct thread_struct *thread) +{ + int res; + enum thread_msg msg = THREAD_MSG__UNDEFINED; + + res = read(thread->pipes.from[0], &msg, sizeof(msg)); + if (res < 0) {Maybe it needs to handle
-EINTR.
Its behaviour should be retry, right?
Since these reads are used multiple times in the code, maybe I'm better off writing a wrapper function handling also EINTR.
Quoted 17 lines
Quoted 17 lines
+ pr_err("threadpool: failed to recv msg from tid=%d: %s\n", +
thread->tid, strerror(errno));
+ return -1;
+ }
+ if (msg !=THREAD_MSG__ACK) {
+ pr_err("threadpool: received unexpected msg from tid=%d:
%s\n",
+thread->tid, thread_msg_tags[msg]);
+ return -1;
+ }tools/perf/util/workqueue/threadpool.c +5
-0+ + pr_debug2("threadpool: received ack from tid=%d\n", thread->tid); + + return 0; +}
<SNIP>
Quoted 40 lines
Quoted 29 lines
tools/perf/util/workqueue/threadpool.c +21
-0+static int __start_threadpool (struct threadpool_struct *pool) +{ + int t, tt, ret = 0, nr_threads = pool->nr_threads; + sigset_t full, mask; + pthread_t handle; + pthread_attr_t attrs; + + sigfillset(&full); + if (sigprocmask(SIG_SETMASK, &full, &mask)) { + pr_err("Failed to block signals on threads start: %s\n", + strerror(errno)); + return -1; + } + + pthread_attr_init(&attrs); + pthread_attr_setdetachstate(&attrs, PTHREAD_CREATE_DETACHED); + + for (t = 0; t < nr_threads; t++) { + struct thread_struct *thread = &pool->threads[t]; + + if (pthread_create(&handle, &attrs, threadpool_thread,thread)) {
+ for (tt = 1; tt < t; tt++)
+terminate_thread(thread); +pr_err("Failed to start threads: %s\n", strerror(errno));
+ ret = -1;
+ gotoout_free_attr;
+ }Quoted 8 lines
+
+ if (wait_thread(thread)) {
+ for (tt = 1; tt <= t; tt++) +terminate_thread(thread); + ret = -1;
+ gotoout_free_attr;
+ }
+ }Isn't it better doing this way?
Quoted 7 lines
for (t = 0; t <
nr_threads; t++) {
pthread_create(t)
}for (t = 0; t <
nr_threads; t++) {
wait_thread(t)
}
I wondered the same thing, but I saw that it was done like that also in Alexey patch, so I kept it like so.
To me, it also looks like it should be not a problem doing as you suggest. It should also be more efficient.
Thanks,
Riccardo
Quoted 18 lines
Thanks,
NamhyungQuoted 13 lines
tools/perf/util/workqueue/threadpool.c +13
-0+ +out_free_attr: + pthread_attr_destroy(&attrs); + + if (sigprocmask(SIG_SETMASK, &mask, NULL)) { + pr_err("Failed to unblock signals on threads start: %s\n", + strerror(errno)); + ret = -1; + } + + return ret; +} +
Re: [RFC PATCH 04/10] perf workqueue: add threadpool execute and wait functions
In reply to Namhyung KimHi Namhyung,
thanks again for the review.
Quoted 47 lines · Namhyung Kim
Quoted 46 lines · Riccardo Mancini
This patch adds:
-execute_in_threadpool: assigns a task to the threads to execute asynchronously.
-wait_threadpool: waits for the task to complete on all threads. Furthermore, testing for these new functions is added.This patch completes the threadpool.
Signed-off-by Riccardo Mancini <[email protected]>---
tools/perf/tests/workqueue.c | 86 ++++++++++++++++++++-tools/perf/util/workqueue/threadpool.c| 103 +++++++++++++++++++++++++tools/perf/util/workqueue/threadpool.h| 5 ++3 files changed, 193 insertions(+), 1 deletion(-)tools/perf/tests/workqueue.c +13
-0@@ -1,13 +1,59 @@ // SPDX-License-Identifier: GPL-2.0 +#include <stdlib.h> #include <linux/kernel.h> +#include <linux/zalloc.h> #include "tests.h" #include "util/debug.h" #include "util/workqueue/threadpool.h" +#define DUMMY_FACTOR 100000 +#define N_DUMMY_WORK_SIZES 7 + struct threadpool_test_args_t { int pool_size; }; +struct test_task { + struct task_struct task; + int n_threads; + int *array; +}; + +/** +* dummy_work - calculates DUMMY_FACTOR * (idx % N_DUMMY_WORK_SIZES) inefficiently
+ *
Quoted 8 lines
tools/perf/tests/workqueue.c line 29
Quoted 5 lines
+ * This function uses modulus to create work items of different sizes.
+ */tools/perf/tests/workqueue.c +3
-0+static void dummy_work (int idx) +{ + int prod = 0;I'm not sure but having 'volatile' would prevent some kind of possible compiler optimizations..
Agreed.
Quoted 57 lines
Quoted 57 lines
+ int k = idx %
N_DUMMY_WORK_SIZES;
+ int i, j;tools/perf/tests/workqueue.c +11
-0+ + for (i = 0; i < DUMMY_FACTOR; i++) + for (j = 0; j < k; j++) + prod ++; + + pr_debug3("dummy: %d * %d = %d\n", DUMMY_FACTOR, k, prod); +} + +static void test_task_fn1 (int tidx,struct task_struct *task) +{ + struct test_task *mtask = container_of(task, struct test_task,task);
tools/perf/tests/workqueue.c +8
-0+ + dummy_work(tidx); + mtask->array[tidx] = tidx+1; +} + +static void test_task_fn2 (int tidx,struct task_struct *task) +{ + struct test_task *mtask = container_of(task, struct test_task,task);
tools/perf/tests/workqueue.c +6
-0+ + dummy_work(tidx); + mtask->array[tidx] = tidx*2; +} + +static int
__threadpool__prepare(struct threadpool_struct **pool, int pool_size)
{
int ret;tools/perf/tests/workqueue.c +15
-0@@ -38,21 +84,59 @@ static int __threadpool__teardown(struct threadpool_struct *pool) return 0; } +static int __threadpool__exec_wait (struct threadpool_struct *pool, + struct task_struct *task) +{ + int ret; + + ret = execute_in_threadpool(pool, task); + TEST_ASSERT_VAL("threadpool execute failure", ret == 0); + TEST_ASSERT_VAL("threadpool is not executing",threadpool_is_busy (pool)); + + ret = wait_threadpool(pool); + TEST_ASSERT_VAL("threadpool wait failure", ret == 0); + TEST_ASSERT_VAL("waited threadpool is not ready",threadpool_is_ready (pool)); + + return 0; +}
Quoted 13 lines
tools/perf/tests/workqueue.c static int __threadpool__teardown(struct threadpool_struct *pool) · line 112
Quoted 11 lines
tools/perf/tests/workqueue.c +6
-1static int __test__threadpool(void *_args) { struct threadpool_test_args_t *args = _args; struct threadpool_struct *pool; - int ret; + int ret, i; + struct test_task task; + + task.task.fn = test_task_fn1; + task.n_threads = args->pool_size; + task.array = calloc(args->pool_size, sizeof(*task.array));Need to check the return value.
Thanks.
Quoted 28 lines
Quoted 28 lines
tools/perf/tests/workqueue.c +16
-0ret = __threadpool__prepare(&pool, args->pool_size); if (ret) return ret; + ret = __threadpool__exec_wait(pool, &task.task); + if (ret) + return ret; + + for (i = 0; i < args->pool_size; i++) + TEST_ASSERT_VAL("failed array check (1)", task.array[i] == i+1 ); + + task.task.fn = test_task_fn2; + + ret = __threadpool__exec_wait(pool, &task.task); + if (ret) + return ret; + + for (i = 0; i < args->pool_size; i++) + TEST_ASSERT_VAL("failed array check (2)", task.array[i] ==2 *i); + ret = __threadpool__teardown(pool); if (ret) return ret;
Quoted 3 lines
tools/perf/tests/workqueue.c static int __threadpool__teardown(struct threadpool_struct *pool) · line 138
Quoted 1 line
tools/perf/tests/workqueue.c +1
-0+ free(task.array);All previous returns will leak it.
Oh, right.
Thanks,
Riccardo
Quoted 3 lines
Thanks,
Namhyung
Arnaldo Carvalho de Melo
maintainerRe: [RFC PATCH 03/10] perf workqueue: add threadpool start and stop functions
In reply to Riccardo ManciniEm Fri, Jul 16, 2021 at 03:53:58PM +0200, Riccardo Mancini escreveu:
Quoted 19 lines · Namhyung Kim
Quoted 17 lines · Riccardo Mancini
Quoted 1 line
+++
b/tools/perf/util/workqueue/threadpool.c[SNIP]
tools/perf/util/workqueue/threadpool.c static void close_pipes(struct thread_struct *thread) · line 129
Quoted 12 lines
tools/perf/util/workqueue/threadpool.c +12
-0+/** +* wait_thread - receive ack from thread +* +* NB: call only from main thread! +*/ +static int wait_thread (struct thread_struct *thread) +{ + int res; + enum thread_msg msg = THREAD_MSG__UNDEFINED; + + res = read(thread->pipes.from[0], &msg, sizeof(msg)); + if (res < 0) {Maybe it needs to handle
-EINTR.
Quoted 3 lines
Its behaviour should be retry, right?
Since these reads are used multiple times in the code, maybe I'm better off writing a wrapper function handling also EINTR.
Take a look at readn():
tools/lib/perf/lib.c
static ssize_t ion(bool is_read, int fd, void *buf, size_t n)
{
void *buf_start = buf;
size_t left = n;
while (left) {
/* buf must be treated as const if !is_read. */
ssize_t ret = is_read ? read(fd, buf, left) :
write(fd, buf, left);
if (ret < 0 && errno == EINTR)
continue;
if (ret <= 0)
return ret;
left -= ret;
buf += ret;
}
BUG_ON((size_t)(buf - buf_start) != n);
return n;
}/*
* Read exactly 'n' bytes or return an error.
*/ssize_t readn(int fd, void *buf, size_t n)
{
return ion(true, fd, buf, n);
}- Arnaldo
Namhyung Kim
maintainerRe: [RFC PATCH 01/10] perf workqueue: threadpool creation and destruction
In reply to Riccardo ManciniQuoted 67 lines · Riccardo Mancini
Hi Namhyung,
thanks for the review.Quoted 33 lines · Namhyung Kim
Hi Riccardo and Arnaldo,
Quoted 29 lines · Arnaldo Carvalho de Melo
Em Tue, Jul 13, 2021 at 02:11:12PM +0200, Riccardo Mancini escreveu:
Quoted 27 lines
The workqueue library is made up by two components:
- threadpool: handles the lifetime of the threads
- workqueue: handles work distribution among the threadsThis first patch introduces the threadpool, starting from its creation and destruction functions.
Thread management is based on the prototype from Alexey:
https://lore.kernel.org/lkml/[email protected]/Each thread in the threadpool executes the same function (aka task) with a different argument tidx.
Threads use a pair of pipes to communicate with the main process. The threadpool is static (all threads will be spawned at the same time). Future work could include making it resizable and adding affinity support (as in Alexey prototype).Suggested-by Alexey Bayduraev <[email protected]>Signed-off-by Riccardo Mancini <[email protected]>--- tools/perf/util/Build | 1 + tools/perf/util/workqueue/Build | 1 + tools/perf/util/workqueue/threadpool.c | 175 +++++++++++++++++++++++++ tools/perf/util/workqueue/threadpool.h | 19 +++ 4 files changed, 196 insertions(+) create mode 100644 tools/perf/util/workqueue/Build create mode 100644 tools/perf/util/workqueue/threadpool.c create mode 100644 tools/perf/util/workqueue/threadpool.h<SNIP>
Quoted 5 lines
Quoted 5 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+ +struct threadpool_struct {Can this be just 'struct threadpool'? I think its descriptive enough:
Quoted 8 lines
Quoted 6 lines
Quoted 6 lines
+ int
nr_threads; /* number of threads in the
pool */
+ structthread_struct*threads; /* array of threads in the
pool */
+ structtask_struct*current_task; /* current executing
function */Does this mean it can only have a single function to run?
Yes.
Quoted 1 line
Why do we need it?
My idea is to separate the workqueue from the actual implementation of the threads. This way, when the function executing on the threadpool ends, the threads are kept alive to execute new work.
By adding this additional layer of abstraction, we can achieve more flexibility. For example, the use-case I have in mind is to recycle the same threadpool for both Alexey's threaded trace and the workqueue.
I don't think this could be easily achieved with just the workqueue since the perf-record threads are not just a task that needs to be executed by they have specific affinities to be respected.
Quoted 1 line
What are your thoughts?
I'm fine with the separation of work(queue) and thread-pool.
I thought the backing thread-pool is general and can handle multiple works at the same time.
The work queue should keep track of works it submitted and their status. We can have multiple workqueues sharing a single thread pool.
Quoted 35 lines
Quoted 18 lines
Quoted 9 lines
Quoted 9 lines
+ enum
threadpool_statusstatus; /* current status of the
pool */tools/perf/util/workqueue/threadpool.c new file +4
-0+}; + +struct thread_struct { +int idx;/* idx of thread in pool- Quoted 8 lines
Quoted 8 lines
Quoted 1 line
threads */
tools/perf/util/workqueue/threadpool.c new file +4
-0+pid_t tid;/* tid of thread */ +struct threadpool_struct *pool;/* parent threadpool */ +struct { +int from[2 ];/* messages from thread (acks) */
+ int to[2]; /* messages to thread
(commands) */Quoted 2 lines
It can be confusing if you think from the main thread. Maybe 'ack' and 'cmd' would be better.
Agreed.
Quoted 11 lines
Quoted 8 lines
tools/perf/util/workqueue/threadpool.c line 32
Quoted 4 lines
tools/perf/util/workqueue/threadpool.c new file +2
-0+ } pipes; +};This one, since we have already a 'struct thread' in
tools/perf, to represent aPERF_RECORD_FORK, perhaps we can call it 'structthreadpool_entry'?I think we can even use 'worker' instead of 'thread' but it requires huge renaming and conflicts so I won't insist on it strongly. :)
Quoted 8 lines
Also, worker internally conflicts with the workqueue's worker, which runs on a (threadpool-)thread.
Another name I had in mind ispool_threadto prevent having too many 'thread' in the name, but it might be confusing.
I thinkthreadpool_entryis fine.I have another question.
In general, when should I use zfree instead of free?
I think the zfree is generally preferable to free. Especially if the pointer can be accessed after free.
Thanks,
Namhyung
Quoted 44 lines · Riccardo Mancini
This patchset introduces a new utility library inside perf/util, which provides a work queue abstraction, which loosely follows the Kernel workqueue API.
The workqueue abstraction is made up by two components:
- threadpool: which takes care of managing a pool of threads. It is inspired by the prototype for threaded trace in perf-record from Alexey:https://lore.kernel.org/lkml/[email protected]/- workqueue: manages a shared queue and provides the workers implementation.
On top of the workqueue, a simple parallel-for utility is implemented which is then showcased in
synthetic-events.c, replacing the previous manual pthread-created threads.Through some experiments with perf bench, I can see how the new workqueue has a higher overhead compared to manual creation of threads, but is able to more effectively partition work among threads, yielding a better result with more threads.
Furthermore, the overhead could be configured by changing thework_size(currently 1), aka the number of dirents that are processed by a thread before grabbing a lock to get the new work item. I experimented with different sizes but, while bigger sizes reduce overhead as expected, they do not scale as well to more threads.I tried to keep the patchset as simple as possible, deferring possible improvements and features to future work.
Naming a few:
- in order to achieve a better performance, we could consider using work-stealing instead of a common queue.
- affinities in the thread pool, as in Alexey prototype for perf-record. Doing so would enable reusing the same threadpool for different purposes (evlist open, threaded trace, synthetic threads), avoiding having to spin up threads multiple times.
- resizable threadpool, e.g. for lazy spawining of threads.@Arnaldo
Since I wanted the workqueue to provide a similar API to the Kernel's workqueue, I followed the naming style I found there, instead of the usualobject__methodstyle that is typically found in perf. Let me know if you'd like me to follow perf style instead.Thanks,
Riccardo
Quoted 11 lines
Riccardo Mancini (10):
perf workqueue: threadpool creation and destruction perf tests: add test for workqueue
perf workqueue: add threadpool start and stop functions perf workqueue: add threadpool execute and wait functions perf workqueue: add sparse annotation header
perf workqueue: introduce workqueue struct
perf workqueue: implement worker thread and management perf workqueue: addqueue_workandflush_workqueuefunctions perf workqueue: add utility to execute a for loop in parallel perf synthetic-events: use workqueueparallel_for
looks great, would it make sense to put this to libperf?
jirka
Quoted 25 lines
tools/perf/tests/Build | 1 + tools/perf/tests/builtin-test.c | 9 + tools/perf/tests/tests.h | 3 + tools/perf/tests/workqueue.c | 453 +++++++++++++++++ tools/perf/util/Build | 1 + tools/perf/util/synthetic-events.c | 131 +++-- tools/perf/util/workqueue/Build | 2 + tools/perf/util/workqueue/sparse.h | 21 + tools/perf/util/workqueue/threadpool.c | 516 ++++++++++++++++++++ tools/perf/util/workqueue/threadpool.h | 29 ++ tools/perf/util/workqueue/workqueue.c | 642 +++++++++++++++++++++++++ tools/perf/util/workqueue/workqueue.h | 38 ++ 12 files changed, 1771 insertions(+), 75 deletions(-) create mode 100644 tools/perf/tests/workqueue.c create mode 100644 tools/perf/util/workqueue/Build create mode 100644 tools/perf/util/workqueue/sparse.h create mode 100644 tools/perf/util/workqueue/threadpool.c create mode 100644 tools/perf/util/workqueue/threadpool.h create mode 100644 tools/perf/util/workqueue/workqueue.c create mode 100644 tools/perf/util/workqueue/workqueue.hSignature · 1 line
2.31.1
Hi Jiri,
Quoted 59 lines · Jiri Olsa
Quoted 46 lines · Riccardo Mancini
This patchset introduces a new utility library inside perf/util, which provides a work queue abstraction, which loosely follows the Kernel workqueue API.
The workqueue abstraction is made up by two components:
- threadpool: which takes care of managing a pool of threads. It is inspired by the prototype for threaded trace in perf-record from Alexey:
https://lore.kernel.org/lkml/[email protected]/
- workqueue: manages a shared queue and provides the workers implementation.On top of the workqueue, a simple parallel-for utility is implemented which is then showcased in
synthetic-events.c, replacing the previous manual pthread-created threads.Through some experiments with perf bench, I can see how the new workqueue has a higher overhead compared to manual creation of threads, but is able to more effectively partition work among threads, yielding a better result with more threads.
Furthermore, the overhead could be configured by changing thework_size(currently 1), aka the number of dirents that are processed by a thread before grabbing a lock to get the new work item. I experimented with different sizes but, while bigger sizes reduce overhead as expected, they do not scale as well to more threads.I tried to keep the patchset as simple as possible, deferring possible improvements and features to future work.
Naming a few:
- in order to achieve a better performance, we could consider using work-stealing instead of a common queue.
- affinities in the thread pool, as in Alexey prototype for perf-record. Doing so would enable reusing the same threadpool for different purposes (evlist open, threaded trace, synthetic threads), avoiding having to spin up threads multiple times.
- resizable threadpool, e.g. for lazy spawining of threads.@Arnaldo
Since I wanted the workqueue to provide a similar API to the Kernel's workqueue, I followed the naming style I found there, instead of the usualobject__methodstyle that is typically found in perf. Let me know if you'd like me to follow perf style instead.Thanks,
RiccardoQuoted 11 lines
Riccardo Mancini (10):
perf workqueue: threadpool creation and destruction perf tests: add test for workqueue
perf workqueue: add threadpool start and stop functions perf workqueue: add threadpool execute and wait functions perf workqueue: add sparse annotation header
perf workqueue: introduce workqueue struct
perf workqueue: implement worker thread and management perf workqueue: addqueue_workandflush_workqueuefunctions perf workqueue: add utility to execute a for loop in parallel perf synthetic-events: use workqueueparallel_for
Quoted 1 line
looks great, would it make sense to put this to libperf?
I don't know about libperf in particular.
The idea is to start using it in perf and, if everything goes well, to put it in lib/ so that everyone interested in it could just include it.
Since I'm looking for other parts where a workqueue could be useful, if you know of some in libperf, I could try having a look at them too.
Riccardo
Quoted 29 lines
jirka
Quoted 25 lines
tools/perf/tests/Build | 1 + tools/perf/tests/builtin-test.c | 9 + tools/perf/tests/tests.h | 3 + tools/perf/tests/workqueue.c | 453 +++++++++++++++++ tools/perf/util/Build | 1 + tools/perf/util/synthetic-events.c | 131 +++-- tools/perf/util/workqueue/Build | 2 + tools/perf/util/workqueue/sparse.h | 21 +
tools/perf/util/workqueue/threadpool.c| 516 ++++++++++++++++++++tools/perf/util/workqueue/threadpool.h| 29 ++ tools/perf/util/workqueue/workqueue.c | 642 +++++++++++++++++++++++++ tools/perf/util/workqueue/workqueue.h | 38 ++12 files changed, 1771 insertions(+), 75 deletions(-)create mode 100644
tools/perf/tests/workqueue.ccreate mode 100644tools/perf/util/workqueue/Buildcreate mode 100644tools/perf/util/workqueue/sparse.hcreate mode 100644tools/perf/util/workqueue/threadpool.ccreate mode 100644tools/perf/util/workqueue/threadpool.hcreate mode 100644tools/perf/util/workqueue/workqueue.ccreate mode 100644tools/perf/util/workqueue/workqueue.hSignature · 1 line
2.31.1