mm/damon: fix stale TLB young-state handling on arm64
Subsystems (2)
PATCH 1/1 mm/damon: fix stale TLB young-state handling on arm64
From: Kunwu Chan <[email protected]>
damon_ptep_mkold() clears the PTE Access Flag so that a later access will set it again and damon_folio_young() can observe it via pte_young().
On arm64, however, ptep_test_and_clear_young() clears AF in the page tables without invalidating the corresponding TLB entry. Subsequent accesses can therefore continue hitting a stale TLB entry without a page table walk. The PTE AF bit stays clear, pte_young() reports false, and DAMON treats the region as unaccessed.
folio_set_idle() does not help here. It updates only software state, and accesses through a stale TLB entry do not clear the idle flag.
As a result, nr_accesses stays low regardless of the real access pattern. DAMOS schemes fail to match, WSS estimation reports zero, and actions like pageout never trigger.
Fix this by switching to ptep_clear_flush_young() and pmdp_clear_flush_young().
On arm64 these perform the required TLB invalidation after clearing AF. The invalidation is deferred, but still sufficient for DAMON's sampling granularity.
On x86, ptep_clear_flush_young() is equivalent to ptep_test_and_clear_young() for base pages, so there is no behavioral change. pmdp_clear_flush_young() additionally performs a flush at PMD level, matching the existing x86 implementation.
On powerpc, riscv, and s390, the clear_flush variants currently map back to test_and_clear implementations, so this patch does not change their behavior.
Reproduced on arm64 (128 CPUs, 7.1.0-rc4):
before:
WSS estimation: 50th percentile error 100% (reported as zero)
apply_interval: schemes never tried
after:
WSS estimation: 50th percentile error 0.08%
apply_interval: passes
--- mm/damon/ops-common.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
mm/damon/ops-common.c +2 -2
@@ -63,7 +63,7 @@ void damon_ptep_mkold(pte_t *pte, struct vm_area_struct *vma, unsigned long addr* device aspects. */ if (likely (pte_present (pteval))) - young |=ptep_test_and_clear_young (vma, addr, pte); + young |=ptep_clear_flush_young (vma, addr, pte); young |=mmu_notifier_clear_young (vma->vm_mm, addr, addr +PAGE_SIZE );if (young)folio_set_young (folio);
@@ -90,7 +90,7 @@ void damon_pmdp_mkold(pmd_t *pmd, struct vm_area_struct *vma, unsigned long addrreturn ;if (likely (pmd_present (pmdval))) - young |=pmdp_test_and_clear_young (vma, addr, pmd); + young |=pmdp_clear_flush_young (vma, addr, pmd); young |=mmu_notifier_clear_young (vma->vm_mm, addr, addr +HPAGE_PMD_SIZE );if (young)folio_set_young (folio);
Signature · 1 line
2.43.0
[email protected]
AI reviewThank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Critical] Replacing pmdp_test_and_clear_young() with pmdp_clear_flush_young() causes kernel panics due…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Critical] Replacing pmdp_test_and_clear_young() with pmdp_clear_flush_young() causes kernel panics due to unaligned addresses and introduces severe performance regressions via synchronous broadcast TLB flushes. --
commit 35d1d30f8c4d83134e4728cac82067099a6e3cab
Author: Kunwu Chan <[email protected]> mm/damon: fix stale TLB young-state handling on arm64
This commit modifies DAMON's page table access functions by replacing test_and_clear_young with clear_flush_young variants for both PTEs and PMDs to resolve stale TLB entry issues on arm64.
Quoted 4 lines
mm/damon/ops-common.c +0
-0
[ ... ]
mm/damon/ops-common.c void damon_pmdp_mkold(pmd_t *pmd, struct vm_area_struct *vma, unsigned long addr · line 93
Quoted 6 lines
mm/damon/ops-common.c +1
-1@@ -90,7 +90,7 @@ void damon_pmdp_mkold(pmd_t *pmd, struct vm_area_struct *vma, unsigned long addrreturn ;if (likely (pmd_present (pmdval))) - young |=pmdp_test_and_clear_young (vma, addr, pmd); + young |=pmdp_clear_flush_young (vma, addr, pmd);
[Severity: Critical]
Does this change risk triggering a kernel panic due to unaligned addresses?
DAMON randomly generates unaligned virtual addresses for sampling via damon_rand(). These unaligned addresses are passed directly to damon_pmdp_mkold() and subsequently to pmdp_clear_flush_young().
On architectures like x86 and s390, and in the generic fallback implementation, pmdp_clear_flush_young() strictly asserts PMD alignment using VM_BUG_ON(address & ~HPAGE_PMD_MASK). Passing DAMON's unaligned sampling address could trigger this assertion.
Additionally, could this introduce a performance regression?
The generic pmdp_clear_flush_young() fallback, which is used by arm64 and riscv since they do not implement __HAVE_ARCH_PMDP_CLEAR_YOUNG_FLUSH, invokes flush_pmd_tlb_range(). This results in a synchronous broadcast TLB flush across the system just for clearing the accessed bit, which seems to conflict with DAMON's low-overhead design constraints.
Quoted 3 lines
mm/damon/ops-common.c +0
-0young |=mmu_notifier_clear_young (vma->vm_mm, addr, addr +HPAGE_PMD_SIZE );if (young)folio_set_young (folio);
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
SeongJae Park
maintainerQuoted 1 line · On Mon, 25 May 2026 22:48:46 +0800 Kunwu Chan <[email protected]> wrote:
From: Kunwu Chan <[email protected]>
Thank you for this great patch!
Quoted 12 lines
damon_ptep_mkold()clears the PTE Access Flag so that a later access will set it again anddamon_folio_young()can observe it viapte_young().On arm64, however,
ptep_test_and_clear_young()clears AF in the page tables without invalidating the corresponding TLB entry. Subsequent accesses can therefore continue hitting a stale TLB entry without a page table walk. The PTE AF bit stays clear,pte_young()reports false, and DAMON treats the region as unaccessed.
Quoted 7 lines
folio_set_idle()does not help here. It updates only software state, and accesses through a stale TLB entry do not clear the idle flag.As a result,
nr_accessesstays low regardless of the real access pattern. DAMOS schemes fail to match, WSS estimation reports zero, and actions like pageout never trigger.
You are correct. Nonetheless, we intentionally designed DAMON in this way, to avoid the performance overhead from the added TLB flushes. Meanwhile, we believed this is ok in terms of the monitoring results accuracy on real world production environment, becasue such environments would have large amount of working set that flushes TLB buffers anyway. We decided to take this way after a measurement [1].
And apparently you found this behavior as problematic on a test environment that having small size of working set. We found a similar issue in DAMON selftest, and we updated the test [2] to simulate the expected real world production environments, rather than changing DAMON.
But, this kind of question is recurring. In addition to the previous discussion, there were a few private inqueries for this issue. And though the real world production environment is the priamry target of DAMON, I understand it is better to support testing environment, too. So, I think it is better to make some changes for this issue, if it doesn't make other problems.
Quoted 8 lines
Fix this by switching to
ptep_clear_flush_young()andpmdp_clear_flush_young().On arm64 these perform the required TLB invalidation after clearing AF. The invalidation is deferred, but still sufficient for DAMON's sampling granularity.
Quoted 8 lines
On x86,
ptep_clear_flush_young()is equivalent toptep_test_and_clear_young()for base pages, so there is no behavioral change.pmdp_clear_flush_young()additionally performs a flush at PMD level, matching the existing x86 implementation.On powerpc, riscv, and s390, the
clear_flushvariants currently map back totest_and_clearimplementations, so this patch does not change their behavior.
This change seems much nicer and might be more optimized than my simple implementation of tlb flush [1] that I tested before.
Quoted 10 lines
Reproduced on arm64 (128 CPUs, 7.1.0-rc4):
before:
WSS estimation: 50th percentile error 100% (reported as zero)
apply_interval: schemes never triedafter:
WSS estimation: 50th percentile error 0.08%
apply_interval: passes
And nice test results. I guess you are referring to the tests in damon-tests? Clarifying the context would be nice.
Also, have you had a chance to measure the performance impact?
So, I'd like to have this change. But, unless we have very clear evidence showing this change is not increasing the performance overhead, I'd prefer making this as an optional feature.
For the user interface, we could add a new sysfs file for the option, say, 'flush_sample_tlb' under 'monitoring_attrs' directory.
For long term, I'm planning [1] to extend the data attributes monitoring feature so that data access becomes just one of the attributes. Once it is done, we could control this tlb flush option using the probes interface.
I was initially thinking about asking Kunwu to wait until the data attributes monitoring extension is done, and add this tlb flush option on top of that. Because, otherwise, we may need to deprecate 'flush_sample_tlb' after the extension is done. But, we will anyway need to deprecate a few interfaces including 'nr_accesses'. Doing the deprecation of 'flush_sample_tlb' together with it shouldn't be huge amount of overhead.
So, unless Kunwu and Lian has other concerns, I'd suggest the 'flush_sample_tlb' path.
[...]
[1] https://lore.kernel.org/[email protected]/
[2] https://lore.kernel.org/[email protected]/
Thanks,
SJ
Quoted 7 lines · SeongJae Park
Quoted 3 lines · On Mon, 25 May 2026 22:48:46 +0800 Kunwu Chan <[email protected]> wrote:
From: Kunwu Chan <[email protected]>
Thank you for this great patch!
Thanks for the reply.
Quoted 21 lines
Quoted 13 lines
damon_ptep_mkold()clears the PTE Access Flag so that a later
access will set it again anddamon_folio_young()can observe it viapte_young().On arm64, however,
ptep_test_and_clear_young()clears AF in the page tables without invalidating the corresponding TLB entry. Subsequent accesses can therefore continue hitting a stale TLB entry without a page table walk. The PTE AF bit stays clear,pte_young()reports false, and DAMON treats the region as unaccessed.Quoted 8 lines
folio_set_idle()does not help here. It updates only software state, and accesses through a stale TLB entry do not clear the idle flag.As a result,
nr_accessesstays low regardless of the real access pattern. DAMOS schemes fail to match, WSS estimation reports zero, and actions like pageout never trigger.
Quoted 7 lines
You are correct. Nonetheless, we intentionally designed DAMON in this way, to avoid the performance overhead from the added TLB flushes. Meanwhile, we believed this is ok in terms of the monitoring results accuracy on real world production environment, becasue such environments would have large amount of working set that flushes TLB buffers anyway. We decided to take this way after a measurement [1].
Understood, thanks for the detailed explanation and context. We were too focused on the testing issue and overlooked the original tradeoff between monitoring accuracy and the overhead of TLB flushing.
Quoted 5 lines
And apparently you found this behavior as problematic on a test environment that having small size of working set. We found a similar issue in DAMON selftest, and we updated the test [2] to simulate the expected real world production environments, rather than changing DAMON.
That makes sense.
The selftest change in [2] also clarified why DAMON normally works fine in production workloads. Large and active workloads naturally exceed TLB coverage, so accessed-bit updates eventually become visible again.
So I agree that adjusting the test instead of forcing TLB flushes in DAMON was the right tradeoff there.
Quoted 6 lines
But, this kind of question is recurring. In addition to the previous discussion, there were a few private inqueries for this issue. And though the real world production environment is the priamry target of DAMON, I understand it is better to support testing environment, too. So, I think it is better to make some changes for this issue, if it doesn't make other problems.
At the same time, I also understand why this keeps coming up. Test environments can differ a lot in workload size, THP usage, VM setup, and hardware configuration, so stable and reproducible testing is not always easy across different systems.
So I agree it makes sense to improve this area as long as it does not add noticeable overhead or cause problems for the normal production case.
Quoted 13 lines
Quoted 13 lines
Fix this by switching to
ptep_clear_flush_young()and
pmdp_clear_flush_young().On arm64 these perform the required TLB invalidation after clearing AF. The invalidation is deferred, but still sufficient for DAMON's sampling granularity.
On x86,
ptep_clear_flush_young()is equivalent toptep_test_and_clear_young()for base pages, so there is no behavioral change.pmdp_clear_flush_young()additionally performs a flush at PMD level, matching the existing x86 implementation.
Quoted 7 lines
Quoted 4 lines
On powerpc, riscv, and s390, the
clear_flushvariants currently map back totest_and_clearimplementations, so this patch does not change their behavior.This change seems much nicer and might be more optimized than my simple implementation of tlb flush [1] that I tested before.
Thanks. Yes, we were mainly trying to reuse the existing arch-optimized helpers as much as possible.
Quoted 7 lines
Quoted 7 lines
Reproduced on arm64 (128 CPUs, 7.1.0-rc4):
before:
WSS estimation: 50th percentile error 100% (reported as zero)
apply_interval: schemes never tried
Quoted 7 lines
Quoted 4 lines
after:
WSS estimation: 50th percentile error 0.08%
apply_interval: passesAnd nice test results. I guess you are referring to the tests in damon-tests? Clarifying the context would be nice.
Yes, those results are from: make -C tools/testing/selftests/damon run_tests on the arm64 test machine mentioned above.
The before/after summary was extracted from the relevant failing tests (sysfs_update_schemes_tried_regions_wss_estimation.py and damos_apply_interval.py) for brevity.
Quoted 1 line
Also, have you had a chance to measure the performance impact?
We haven't done detailed performance measurements yet, but we can try to collect some numbers for the flush overhead on a few different setups.
Quoted 4 lines
So, I'd like to have this change. But, unless we have very clear evidence showing this change is not increasing the performance overhead, I'd prefer making this as an optional feature.
We agree that making it optional sounds safer unless we have solid evidence showing the overhead is negligible. Keeping the current default behavior for production workloads also makes sense to me.
Quoted 3 lines
For the user interface, we could add a new sysfs file for the option, say, '
flush_sample_tlb' under 'monitoring_attrs' directory.
The proposed 'flush_sample_tlb' interface under monitoring_attrs sounds reasonable to me as well.
Quoted 11 lines
For long term, I'm planning [1] to extend the data attributes monitoring feature so that data access becomes just one of the attributes. Once it is done, we could control this tlb flush option using the probes interface.
I was initially thinking about asking Kunwu to wait until the data attributes monitoring extension is done, and add this tlb flush option on top of that. Because, otherwise, we may need to deprecate '
flush_sample_tlb' after the extension is done. But, we will anyway need to deprecate a few interfaces including 'nr_accesses'. Doing the deprecation of 'flush_sample_tlb' together with it shouldn't be huge amount of overhead.
Quoted 2 lines
So, unless Kunwu and Lian has other concerns, I'd suggest the '
flush_sample_tlb' path.
Yes, we also think deprecating it later together with interfaces such as 'nr_accesses' should be manageable once the data attributes monitoring extension is ready.
Thanks, Kunwu
Quoted 9 lines
[...]
[1] https://lore.kernel.org/[email protected]/
[2] https://lore.kernel.org/[email protected]/Thanks,
SJ
SeongJae Park
maintainerQuoted 5 lines · On Tue, 26 May 2026 08:57:32 +0000 "Kunwu Chan" <[email protected]> wrote:
Quoted 2 lines · SeongJae Park
On Mon, 25 May 2026 22:48:46 +0800 Kunwu Chan <[email protected]> wrote:
[...]
Quoted 13 lines
Quoted 6 lines
Quoted 6 lines
Reproduced on arm64 (128 CPUs, 7.1.0-rc4):
before:
WSS estimation: 50th percentile error 100% (reported as zero)
apply_interval: schemes never triedQuoted 7 lines
Quoted 4 lines
after:
WSS estimation: 50th percentile error 0.08%
apply_interval: passesAnd nice test results. I guess you are referring to the tests in damon-tests? Clarifying the context would be nice.
Quoted 6 lines
Yes, those results are from: make -C
tools/testing/selftests/damonrun_testson the arm64 test machine mentioned above.The before/after summary was extracted from the relevant failing tests (
sysfs_update_schemes_tried_regions_wss_estimation.pyanddamos_apply_interval.py) for brevity.
Thank you for clarifying!
wss_estimation increases its working set size up to 160 MiB for this issue. Seems your test machine has large TLB buffer. I think we should decide the limit based on the real running system configuration and apply similar approach to other tests including the apply_interval.
For out-of-tree tests, we may better to provide a guideline, too. E.g., run this sort of test program with this DAMON config to find the reliable test working set size on your setup.
Quoted 13 lines
Quoted 2 lines
Also, have you had a chance to measure the performance impact?
We haven't done detailed performance measurements yet, but we can try to collect some numbers for the flush overhead on a few different setups.
Quoted 4 lines
So, I'd like to have this change. But, unless we have very clear evidence showing this change is not increasing the performance overhead, I'd prefer making this as an optional feature.
We agree that making it optional sounds safer unless we have solid evidence showing the overhead is negligible. Keeping the current default behavior for production workloads also makes sense to me.
Quoted 5 lines
Quoted 3 lines
For the user interface, we could add a new sysfs file for the option, say, '
flush_sample_tlb' under 'monitoring_attrs' directory.The proposed '
flush_sample_tlb' interface undermonitoring_attrssounds reasonable to me as well.
I was thinking this again. I still want DAMON to be easy to test. But, is this making tests that difficult? Users could increase the test working set size. I'm not very sure that is too diifficult to add new optional feature. Meanwhille, adding an optional feature for only test might make users be confused. DAMON usage might also be diverged and add maintenance burdens.
So, now I think another option is improving the documentation. It shouldd clearly explain how and why DAMON does not flush TLB and what is the expected problems (in tests) and recommendation. In this option, we should also update existing DAMON tests to be reliable and aligned with the documented recommendation. If we find it becomes a problem on testing even after applying the recommendation, or on production, we can revisit.
Regardless of the decision about the optional feature in DAMON, I think such documentation and tests improvement should be made.
Maybe I'm biased, so any input would be appreicatedd. What do you think, Kunwu and Lian?
Thanks,
SJ
[...]
Hi SJ,
[...]
Quoted 11 lines
Thank you for clarifying!
wss_estimationincreases its working set size up to 160 MiB for this issue. Seems your test machine has large TLB buffer. I think we should decide the limit based on the real running system configuration and apply similar approach to other tests including theapply_interval.For out-of-tree tests, we may better to provide a guideline, too. E.g., run this sort of test program with this DAMON config to find the reliable test working set size on your setup.
Thanks for the detailed analysis.
We agree that improving the documentation and aligning the tests is a better direction for now. Adding an optional feature mainly for testing could confuse users and create additional maintenance burden.
We've already done some selftests:
https://lore.kernel.org/damon/[email protected]/
[...]
Quoted 14 lines
Quoted 1 line
I was thinking this again. I still want DAMON to be easy to test. But, is this making tests that difficult? Users could increase the test working set size. I'm not very sure that is too diifficult to add new optional feature. Meanwhille, adding an optional feature for only test might make users be confused. DAMON usage might also be diverged and add maintenance burdens.
So, now I think another option is improving the documentation. It shouldd clearly explain how and why DAMON does not flush TLB and what is the expected problems (in tests) and recommendation. In this option, we should also update existing DAMON tests to be reliable and aligned with the documented recommendation. If we find it becomes a problem on testing even after applying the recommendation, or on production, we can revisit.
Quoted 6 lines
Regardless of the decision about the optional feature in DAMON, I think such documentation and tests improvement should be made.
Maybe I'm biased, so any input would be appreicatedd. What do you think, Kunwu and Lian?
We think that makes sense. Explaining the rationale for not flushing TLB, the limitations this can introduce for tests, and recommendations for choosing reliable test working set sizes would be helpful.
If we later find cases where the documented recommendations are still insufficient, we can revisit more intrusive approaches.
I'd be happy to help with the documentation work if needed. Please let me know if you'd like me to prepare a draft patch.
Thanks,
Kunwu
Quoted 5 lines
Thanks,
SJ[...]
SeongJae Park
maintainerOn Sun, 31 May 2026 12:16:49 +0000 "Kunwu Chan" <[email protected]> wrote:
[...]
Quoted 3 lines
We think that makes sense. Explaining the rationale for not flushing TLB, the limitations this can introduce for tests, and recommendations for choosing reliable test working set sizes would be helpful.
Thank you for kindly accepting my suggestion.
Quoted 6 lines
If we later find cases where the documented recommendations are still insufficient, we can revisit more intrusive approaches.
I'd be happy to help with the documentation work if needed. Please let me know if you'd like me to prepare a draft patch.
Yes, please feel free to send a patch! :)
Thanks,
SJ
[...]
Quoted 12 lines · SeongJae Park
On Sun, 31 May 2026 12:16:49 +0000 "Kunwu Chan" <[email protected]> wrote:
[...]
Quoted 5 lines
We think that makes sense. Explaining the rationale for not flushing TLB,
the limitations this can introduce for tests, and recommendations for choosing reliable test working set sizes would be helpful.Thank you for kindly accepting my suggestion.
Thanks, SJ.
Quoted 9 lines
Quoted 7 lines
If we later find cases where the documented recommendations are still
insufficient, we can revisit more intrusive approaches.I'd be happy to help with the documentation work if needed. Please let me know if you'd like me to prepare a draft patch.
Yes, please feel free to send a patch! :)
Will send it once I have something ready.
Thanks, Kunwu
Quoted 5 lines
Thanks,
SJ[...]