perf: High-order AUX allocations
Alexander Shishkin
reviewercover perf: High-order AUX allocations
Hi Peter,
This supersedes [1]. Since there is no reason not to always go with high order allocations, we simply change the AUX allocator to optimistically do that with a fallback to lower orders. And since this is not an opt-in any more, no tool changes.
[1] https://marc.info/?l=linux-kernel&m=155005848007143
Alexander Shishkin (1):
perf: Optimistically use high order allocations for AUX buffers
kernel/events/ring_buffer.c | 32 +++++++++++++++----------------- 1 file changed, 15 insertions(+), 17 deletions(-)
Signature · 1 line
2.20.1
Alexander Shishkin
reviewerPATCH 1/1 perf: Optimistically use high order allocations for AUX buffers
Currently, the AUX buffer allocator will use high-order allocations for PMUs that don't support hardware scatter-gather chaining to ensure large contiguous blocks of pages, and always use an array of single pages otherwise.
There is, however, a tangible performance benefit in using larger chunks of contiguous memory even in the latter case, that comes from not having to fetch the next page's address at every page boundary. In particular, a task running under Intel PT on an Atom CPU shows 1.5%-2% less runtime penalty with a single multi-page output region in snapshot mode (no PMI) than with multiple single-page output regions, from ~6% down to ~4%. For the snapshot mode it does make a difference as it is intended to run over long periods of time.
For this reason, change the allocation policy to always optimistically start with the highest possible order when allocating pages for the AUX buffer, desceding until the allocation succeeds or order zero allocation fails.
--- kernel/events/ring_buffer.c | 32 +++++++++++++++----------------- 1 file changed, 15 insertions(+), 17 deletions(-)
kernel/events/ring_buffer.c +15 -17
@@ -598,29 +598,27 @@ int rb_alloc_aux(struct ring_buffer *rb, struct perf_event *event, {bool overwrite = !(flags &RING_BUFFER_WRITABLE );int node = (event->cpu ==-1 ) ?-1 :cpu_to_node (event->cpu); -int ret = -ENOMEM , max_order =0 ; +int ret = -ENOMEM , max_order;if (!has_aux (event))return -EOPNOTSUPP ; -if (event->pmu->capabilities &PERF_PMU_CAP_AUX_NO_SG ) { -/* -* We need to start with the max_order that fits in nr_pages, -* not the other way around, hence ilog2() and not get_order. -*/ - max_order =ilog2 (nr_pages); +/* +* We need to start with the max_order that fits in nr_pages, +* not the other way around, hence ilog2() and not get_order. +*/ + max_order =ilog2 (nr_pages); -/* -* PMU requests more than one contiguous chunks of memory -* for SW double buffering -*/ -if ((event->pmu->capabilities &PERF_PMU_CAP_AUX_SW_DOUBLEBUF ) && - !overwrite) { -if (!max_order) -return -EINVAL ; +/* +* PMU requests more than one contiguous chunks of memory +* for SW double buffering +*/ +if ((event->pmu->capabilities &PERF_PMU_CAP_AUX_SW_DOUBLEBUF ) && + !overwrite) { +if (!max_order) +return -EINVAL ; - max_order--; - } + max_order--; } rb->aux_pages = kcalloc_node(nr_pages, sizeof(void *), GFP_KERNEL,
Signature · 1 line
2.20.1
Alexander Shishkin
reviewerRe: [PATCH v1 1/1] perf: Optimistically use high order allocations for AUX buffers
Quoted 14 lines · Alexander Shishkin
Currently, the AUX buffer allocator will use high-order allocations for PMUs that don't support hardware scatter-gather chaining to ensure large contiguous blocks of pages, and always use an array of single pages otherwise.
There is, however, a tangible performance benefit in using larger chunks of contiguous memory even in the latter case, that comes from not having to fetch the next page's address at every page boundary. In particular, a task running under Intel PT on an Atom CPU shows 1.5%-2% less runtime penalty with a single multi-page output region in snapshot mode (no PMI) than with multiple single-page output regions, from ~6% down to ~4%. For the snapshot mode it does make a difference as it is intended to run over long periods of time.
Quoted 6 lines
For this reason, change the allocation policy to always optimistically start with the highest possible order when allocating pages for the AUX buffer, desceding until the allocation succeeds or order zero allocation fails.
Signed-off-by Alexander Shishkin <[email protected]>
Does anybody want to pick this up?
Thanks,
--
Alex
tip-bot for Alexander Shishkin
merge bot[tip:perf/urgent] perf/ring_buffer: Use high order allocations for AUX buffers optimistically
In reply to Alexander Shishkinperf/ring_buffer: Use high order allocations for AUX buffers optimistically
Commit-ID: 5768402fd9c6e872252b5268ad85e3fbae4fe26b Gitweb: https://git.kernel.org/tip/5768402fd9c6e872252b5268ad85e3fbae4fe26b Author: Alexander Shishkin <[email protected]> AuthorDate: Fri, 15 Feb 2019 13:47:27 +0200 Committer: Ingo Molnar <[email protected]> CommitDate: Sat, 9 Mar 2019 14:10:30 +0100
perf/ring_buffer: Use high order allocations for AUX buffers optimistically
Currently, the AUX buffer allocator will use high-order allocations for PMUs that don't support hardware scatter-gather chaining to ensure large contiguous blocks of pages, and always use an array of single pages otherwise.
There is, however, a tangible performance benefit in using larger chunks of contiguous memory even in the latter case, that comes from not having to fetch the next page's address at every page boundary. In particular, a task running under Intel PT on an Atom CPU shows 1.5%-2% less runtime penalty with a single multi-page output region in snapshot mode (no PMI) than with multiple single-page output regions, from ~6% down to ~4%. For the snapshot mode it does make a difference as it is intended to run over long periods of time.
For this reason, change the allocation policy to always optimistically start with the highest possible order when allocating pages for the AUX buffer, desceding until the allocation succeeds or order zero allocation fails.
--- kernel/events/ring_buffer.c | 32 +++++++++++++++----------------- 1 file changed, 15 insertions(+), 17 deletions(-)
kernel/events/ring_buffer.c +15 -17
@@ -598,29 +598,27 @@ int rb_alloc_aux(struct ring_buffer *rb, struct perf_event *event, {bool overwrite = !(flags &RING_BUFFER_WRITABLE );int node = (event->cpu ==-1 ) ?-1 :cpu_to_node (event->cpu); -int ret = -ENOMEM , max_order =0 ; +int ret = -ENOMEM , max_order;if (!has_aux (event))return -EOPNOTSUPP ; -if (event->pmu->capabilities &PERF_PMU_CAP_AUX_NO_SG ) { -/* -* We need to start with the max_order that fits in nr_pages, -* not the other way around, hence ilog2() and not get_order. -*/ - max_order =ilog2 (nr_pages); +/* +* We need to start with the max_order that fits in nr_pages, +* not the other way around, hence ilog2() and not get_order. +*/ + max_order =ilog2 (nr_pages); -/* -* PMU requests more than one contiguous chunks of memory -* for SW double buffering -*/ -if ((event->pmu->capabilities &PERF_PMU_CAP_AUX_SW_DOUBLEBUF ) && - !overwrite) { -if (!max_order) -return -EINVAL ; +/* +* PMU requests more than one contiguous chunks of memory +* for SW double buffering +*/ +if ((event->pmu->capabilities &PERF_PMU_CAP_AUX_SW_DOUBLEBUF ) && + !overwrite) { +if (!max_order) +return -EINVAL ; - max_order--; - } + max_order--; } rb->aux_pages = kcalloc_node(nr_pages, sizeof(void *), GFP_KERNEL,