Skip to content

RFC perf: add workqueue library and use it in synthetic-events

Riccardo Mancini linux-perf-users lkml 4 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v3 RFC Latest15 patches 8 repliesIn review
  2. v2 RFC posting 2/210 patches 11 repliesSuperseded
  3. v2 RFC posting 1/20 of 10 patches 0 repliesSuperseded
Earlier versions · 1
VersionPostedPatchesRepliesStatus
v1 RFC10 patches22 repliesSuperseded

Changes in v3 from the cover letter

- improved separation of threadpool and threadpool_entry method
- replaced shared workqueue with per-thread workqueue. This should improve the performance on big machines (Jiri noticed in his experiments a significant performance degradation after 15 threads with the shared queue).
- improved error reporting in both threadpool and workqueue
- added lazy spinup of threads in workqueue [9/15]
- added global workqueue [10/15]
- setup global workqueue in perf record, top and synthesize bench [12-14/15] and used in in synthetic events

Patches in v3 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/15perf workqueue: threadpool creation and destructiontools/perf/util/Build tools/perf/util/workqueue/Build tools/perf/util/workqueue/threadpool.c +1 more+231 -00 replies
2/15perf tests: add test for workqueuetools/perf/tests/Build tools/perf/tests/builtin-test.c tools/perf/tests/tests.h +1 more+131 -00 replies
3/15perf workqueue: add threadpool start and stop functionstools/perf/tests/workqueue.c tools/perf/util/workqueue/threadpool.c tools/perf/util/workqueue/threadpool.h+347 -20 replies
4/15perf workqueue: add threadpool execute and wait functionstools/perf/tests/workqueue.c tools/perf/util/workqueue/threadpool.c tools/perf/util/workqueue/threadpool.h+183 -10 replies
5/15tools: add sparse context/locking annotations in compiler-types.htools/include/linux/compiler_types.h+18 -00 replies
6/15perf workqueue: introduce workqueue structtools/perf/tests/workqueue.c tools/perf/util/workqueue/Build tools/perf/util/workqueue/workqueue.c +1 more+317 -02 replies
7/15perf workqueue: implement worker thread and managementtools/perf/util/workqueue/workqueue.c+374 -21 reply
8/15perf workqueue: add queue_work and flush_workqueue functionstools/perf/tests/workqueue.c tools/perf/util/workqueue/workqueue.c tools/perf/util/workqueue/workqueue.h+254 -22 replies
9/15perf workqueue: spinup threads when neededtools/perf/util/workqueue/workqueue.c+38 -160 replies
10/15perf workqueue: create global workqueuetools/perf/util/workqueue/workqueue.c tools/perf/util/workqueue/workqueue.h+51 -00 replies
11/15perf workqueue: add utility to execute a for loop in paralleltools/perf/tests/workqueue.c tools/perf/util/workqueue/workqueue.c tools/perf/util/workqueue/workqueue.h+231 -00 replies
12/15perf record: setup global workqueuetools/perf/builtin-record.c+15 -00 replies
13/15perf top: setup global workqueuetools/perf/builtin-top.c+16 -00 replies
14/15perf test/synthesis: setup global workqueuetools/perf/bench/synthesize.c+14 -20 replies
15/15perf synthetic-events: use workqueue parallel_fortools/perf/bench/synthesize.c tools/perf/builtin-kvm.c tools/perf/builtin-record.c +5 more+56 -1120 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v3 yet.

Read the v3 thread

Cover letter [RFC PATCH v3 00/15] perf: add workqueue library and use it in synthetic-events

Changes in v3:
- improved separation of threadpool and threadpool_entry method
- replaced shared workqueue with per-thread workqueue. This should improve the performance on big machines (Jiri noticed in his experiments a significant performance degradation after 15 threads with the shared queue).
- improved error reporting in both threadpool and workqueue
- added lazy spinup of threads in workqueue [9/15]
- added global workqueue [10/15]
- setup global workqueue in perf record, top and synthesize bench [12-14/15] and used in in synthetic events

v2: https://lore.kernel.org/lkml/[email protected]/

    https://lore.kernel.org/lkml/4f0cd6c8e77c0b4f4d4b8d553a7032757b976e61.1627657061.git.rickyman7@gmail.com/

(something went wrong when sending it and the cover letter has a wrong Message-Id)

Changes in v2:
- rename threadpool_struct and its functions to adhere to naming style
- use ERR_PTR instead of returning NULL
- add *__strerror functions, removing pr_err from library code
- wait for threads after creation of all threads, instead of waiting after each creation
- use intention-revealing macros in test code instead of 0 and -1
- use readn/writen functions

v1: https://lkml.kernel.org/lkml/[email protected]/

This patchset introduces a new utility library inside perf/util, which provides a work queue abstraction, which follows the Kernel workqueue API.

The workqueue abstraction is made up by two components:
- threadpool: which takes care of managing a pool of threads. It is inspired by the prototype for threaded trace in perf-record from Alexey:
https://lore.kernel.org/lkml/[email protected]/
- workqueue: manages the workers in the threadpool and assigns the work items to the thread-local queues.

On top of the workqueue, a simple parallel-for utility is implemented which is then showcased in synthetic-events.c, replacing the previous manual pthread-created threads.

Through some experiments with perf bench, I can see how the new workqueue has a slightly higher overhead compared to manual creation of threads, but is able to more effectively partition work among threads, yielding better results overall (see the last patch for benchmark detaisl on my machine).
Since I'm not able to test it on bigger machine, it would be helpful if someone could also test it and report back his results (thanks to Jiri and Arnaldo who have already helped me by doing some tests).

Furthermore, the overhead could be reduced by changing the work_size (currently 1), aka the number of dirents that are processed by a thread before grabbing a lock to get the new work item. I experimented with different sizes but, while bigger sizes reduce overhead as expected, they do not scale as well to more threads.

Soon I will also send another patchset applying the workqueue to evlist operations (open, enable, disable, close).

Thanks,
Riccardo

Riccardo Mancini (15):
perf workqueue: threadpool creation and destruction perf tests: add test for workqueue
perf workqueue: add threadpool start and stop functions perf workqueue: add threadpool execute and wait functions
tools: add sparse context/locking annotations in compiler-types.h perf workqueue: introduce workqueue struct
perf workqueue: implement worker thread and management perf workqueue: add queue_work and flush_workqueue functions perf workqueue: spinup threads when needed
perf workqueue: create global workqueue
perf workqueue: add utility to execute a for loop in parallel perf record: setup global workqueue
perf top: setup global workqueue
perf test/synthesis: setup global workqueue
perf synthetic-events: use workqueue parallel_for

 tools/include/linux/compiler_types.h   |  18 +
 tools/perf/bench/synthesize.c          |  28 +-
 tools/perf/builtin-kvm.c               |   2 +-
 tools/perf/builtin-record.c            |  18 +-
 tools/perf/builtin-top.c               |  19 +-
 tools/perf/builtin-trace.c             |   3 +-
 tools/perf/tests/Build                 |   1 +
 tools/perf/tests/builtin-test.c        |   9 +
 tools/perf/tests/mmap-thread-lookup.c  |   2 +-
 tools/perf/tests/tests.h               |   3 +
 tools/perf/tests/workqueue.c           | 453 +++++++++++++
 tools/perf/util/Build                  |   1 +
 tools/perf/util/synthetic-events.c     | 135 ++--
 tools/perf/util/synthetic-events.h     |   8 +-
 tools/perf/util/workqueue/Build        |   2 +
 tools/perf/util/workqueue/threadpool.c | 619 +++++++++++++++++
 tools/perf/util/workqueue/threadpool.h |  41 ++
 tools/perf/util/workqueue/workqueue.c  | 901 +++++++++++++++++++++++++
 tools/perf/util/workqueue/workqueue.h  | 104 +++
 19 files changed, 2253 insertions(+), 114 deletions(-)
 create mode 100644 tools/perf/tests/workqueue.c
 create mode 100644 tools/perf/util/workqueue/Build
 create mode 100644 tools/perf/util/workqueue/threadpool.c
 create mode 100644 tools/perf/util/workqueue/threadpool.h
 create mode 100644 tools/perf/util/workqueue/workqueue.c
 create mode 100644 tools/perf/util/workqueue/workqueue.h
Signature · 1 line
2.31.1