mm/damon/core: Performance optimizations for the kdamond hot path
Subsystems (2)
v2 cover mm/damon/core: Performance optimizations for the kdamond hot path
Hello,
This patch series provides two performance optimizations for the DAMON core, specifically targeting the hot paths in kdamond.
The first patch optimizes kdamond_apply_schemes() by inverting the loop order. By iterating over schemes first and regions second, we can evaluate scheme-level invariants (like activation status and quotas) once per scheme rather than for every single region. This significantly reduces CPU overhead when multiple schemes are present or when quotas are reached.
The second patch eliminates a hardware integer division in damon_max_nr_accesses() by using the pre-cached aggr_samples value. Since this function is called once per region per sampling interval, removing the division provides a measurable reduction in CPU cycles spent in the access rate update path.
Changes from v1:
- Use min_t(unsigned long, ...) in damon_max_nr_accesses() to satisfy checkpatch warnings and improve readability.
Josh Law (2):
mm/damon/core: optimize kdamond_apply_schemes() by inverting scheme
and region loops
mm/damon/core: eliminate hot-path integer division in
damon_max_nr_accesses()
include/linux/damon.h | 3 +- mm/damon/core.c | 68 ++++++++++++++++++------------------------- 2 files changed, 29 insertions(+), 42 deletions(-)
Signature · 1 line
2.43.0
PATCH v2 1/2 mm/damon/core: optimize kdamond_apply_schemes() by inverting scheme and region loops
Currently, kdamond_apply_schemes() iterates over all targets, then over all regions, and finally calls damon_do_apply_schemes() which iterates over all schemes. This nested structure causes scheme-level invariants (such as time intervals, activation status, and quota limits) to be evaluated inside the innermost loop for every single region.
If a scheme is inactive, has not reached its apply interval, or has already fulfilled its quota (quota->charged_sz >= quota->esz), the kernel still needlessly iterates through thousands of regions only to repeatedly evaluate these same scheme-level conditions and continue.
This patch inlines damon_do_apply_schemes() into kdamond_apply_schemes() and inverts the loop ordering. It now iterates over schemes on the outside, and targets/regions on the inside.
This allows the code to evaluate scheme-level limits once per scheme. If a scheme's quota is met or it is inactive, we completely bypass the O(Targets * Regions) inner loop for that scheme. This drastically reduces unnecessary branching, cache thrashing, and CPU overhead in the kdamond hot path.
--- mm/damon/core.c | 72 +++++++++++++++++++++---------------------------- 1 file changed, 30 insertions(+), 42 deletions(-)
mm/damon/core.c +30 -42
@@ -2112,40 +2112,6 @@ static void damos_apply_scheme(struct damon_ctx *c, struct damon_target *t,damos_update_stat (s, sz, sz_applied, sz_ops_filter_passed); } -static void damon_do_apply_schemes (struct damon_ctx *c, -struct damon_target *t, -struct damon_region *r) -{ -struct damos *s; - -damon_for_each_scheme (s, c) { -struct damos_quota *quota = &s->quota; - -if (time_before (c->passed_sample_intervals, s->next_apply_sis)) -continue ; - -if (!s->wmarks.activated) -continue ; - -/* Check the quota */ -if (quota->esz && quota->charged_sz >= quota->esz) -continue ; - -if (damos_skip_charged_region (t, r, s, c->min_region_sz)) -continue ; - -if (s->max_nr_snapshots && - s->max_nr_snapshots <= s->stat.nr_snapshots) -continue ; - -if (damos_valid_target (c, r, s)) -damos_apply_scheme (c, t, r, s); - -if (damon_is_last_region (r, t)) - s->stat.nr_snapshots++; - } -} -/* * damon_feed_loop_next_input() - get next input to achieve a target score. * @last_input The last input.
@@ -2494,17 +2460,39 @@ static void kdamond_apply_schemes(struct damon_ctx *c)return ;mutex_lock (&c->walk_control_lock); -damon_for_each_target (t, c) { -if (c->ops.target_valid && c->ops.target_valid (t) == false) -continue ; - -damon_for_each_region (r, t) -damon_do_apply_schemes (c, t, r); - } -damon_for_each_scheme (s, c) { +struct damos_quota *quota = &s->quota; +if (time_before (c->passed_sample_intervals, s->next_apply_sis))continue ; + +if (!s->wmarks.activated) +continue ; + +damon_for_each_target (t, c) { +if (c->ops.target_valid && c->ops.target_valid (t) == false) +continue ; + +damon_for_each_region (r, t) { +/* Check the quota */ +if (quota->esz && quota->charged_sz >= quota->esz) + goto next_scheme; + +if (s->max_nr_snapshots && + s->max_nr_snapshots <= s->stat.nr_snapshots) + goto next_scheme; + +if (damos_skip_charged_region (t, r, s, c->min_region_sz)) +continue ; + +if (damos_valid_target (c, r, s)) +damos_apply_scheme (c, t, r, s); + +if (damon_is_last_region (r, t)) + s->stat.nr_snapshots++; + } + } +next_scheme:damos_walk_complete (c, s);damos_set_next_apply_sis (s, c); s->last_applied =NULL ;
Signature · 1 line
2.34.1
PATCH v2 2/2 mm/damon/core: eliminate hot-path integer division in damon_max_nr_accesses()
In reply to Josh LawHardware integer division is slow. The function damon_max_nr_accesses(), which is called very frequently (e.g., once per region per sample interval inside damon_update_region_access_rate), performs an integer
division: attrs->aggr_interval / attrs->sample_interval.
However, the struct damon_attrs already caches this exact ratio in the internal field aggr_samples (since earlier commits). We can eliminate the hardware division in the hot path by simply returning aggr_samples.
This significantly reduces the CPU cycle overhead of updating the access rates for thousands of regions.
--- include/linux/damon.h | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
include/linux/damon.h +1 -2
@@ -960,8 +960,7 @@ static inline bool damon_target_has_pid(const struct damon_ctx *ctx)static inline unsigned int damon_max_nr_accesses (const struct damon_attrs *attrs) {/* {aggr,sample}_interval are unsigned long, hence could overflow */ -return min (attrs->aggr_interval / attrs->sample_interval, - (unsigned long )UINT_MAX ); + return min_t(unsigned long, attrs->aggr_samples, UINT_MAX); }
Signature · 1 line
2.34.1
SeongJae Park
maintainerI show you already posted v3 [1] of this series. So I'm skipping this version.
[1] https://lore.kernel.org/[email protected]
Thanks,
SJ
Quoted 28 lines · On Sun, 22 Mar 2026 21:36:29 +0000 Josh Law <[email protected]> wrote:
Hello,
This patch series provides two performance optimizations for the DAMON core, specifically targeting the hot paths in kdamond.
The first patch optimizes
kdamond_apply_schemes()by inverting the loop order. By iterating over schemes first and regions second, we can evaluate scheme-level invariants (like activation status and quotas) once per scheme rather than for every single region. This significantly reduces CPU overhead when multiple schemes are present or when quotas are reached.The second patch eliminates a hardware integer division in
damon_max_nr_accesses()by using the pre-cachedaggr_samplesvalue. Since this function is called once per region per sampling interval, removing the division provides a measurable reduction in CPU cycles spent in the access rate update path.Changes from v1:
- Usemin_t(unsigned long, ...)indamon_max_nr_accesses()to satisfy checkpatch warnings and improve readability.Josh Law (2):
mm/damon/core: optimizekdamond_apply_schemes()by inverting scheme
and region loops
mm/damon/core: eliminate hot-path integer division in
damon_max_nr_accesses()
Quoted 6 lines
include/linux/damon.h | 3 +- mm/damon/core.c | 68 ++++++++++++++++++------------------------- 2 files changed, 29 insertions(+), 42 deletions(-)Signature · 1 line
2.43.0
Sent using hkml (https://github.com/sjp38/hackermail)