Notes from February 12, 2026
Hi everybody,
Here are the notes from the last Linux Memory Hotness and Promotion call that happened on Thursday, February 12. Thanks to everybody who was involved!
These notes are intended to bring people up to speed who could not attend the call as well as keep the conversation going in between meetings.
----->o-----
Bharata updated on the status of v5 of his patch series from a couple weeks ago with two modes of operation. Based on discussion with people are LPC, he proposed two mechanisms to track hotness that would have vastly different memory footprint requirements, one as little as one byte per page. He posted benchmark numbers over a series of updates upstream. He found a clear advantage of hot page promotion when top tier memory was not saturated for both of these modes of operation.
However, when there is top-tier memory pressure (when the benchmark working set spills over into CXL memory), then we tend to demote and promote simultaneously as expected. Interestingly, he has not observed promotion in these cases actually being helpful; in fact, in some cases it actually turns out to hurt. He asked for feedback on tunings for when the demotion case can actually help this scenario. Note in one of these benchmarking scenarios, however, that the benchmark would perform random access.
Gregory suggested that we may want to track the number of demotions and promotions; when the rate of demotions equals the rate of promotions then we should reduce the number of promotions. Bharata noted that there was ratelimiting on promotions but not considering the current rate of demotions. In this case, Gregory said, we are seeing promotions drive demotions and there will be an endless loop with these workload characteristics that will impact performance.
Gregory's idea was that if we promote 1,000 pages and then a few seconds later we demote 1,000 pages, then this is an indication of churn. If this is consistent, we should back off. In an ideal scenario, we would be doing some proactive demotion from top tier so there is room to promote. He wanted to see data on the rate of promotions and rate of demotions over time for these bnechmarks. This may reveal an indicator that we can use as a heuristic for a back off mechanism.
I asked if we need a back off mechanism or we actually need a fairness mechanism, the worry was that we would find churn in promotion and demotion, then back off, then start promoting and demoting again at the same rate only to rinse and repeat. Gregory noted that this will continue to consume precious bandwidth on the device so any churn will naturally impact the rest of the system.
Wei Xu thought that this was more about the sorting of the cold memory: we need to rank the page coldness across tiers because what we promoted may not necessarily be hotter than what we promoted. One possible approach is to use a time dimension where we refuse to demote any memory that is not cold enough (like MGLRU's min_ttl). Gregory noted this is a function of the call that we use to promote the memory which, today, is just a call to migrate_pages(); we could pass gfp flags to specify what to do if the promotion node is out of memory. If we are calling direct reclaim, that's probably an indication that we are not aging off inactive pages and we may be forcing demotion of active pages.
Bharata's current approach uses migrate_misplaced_folio() which unfortuantely does not give a migration context but does wake up kswapd if the top tier is near an oom condition. There is no gfp flags being passed, however. We may want to extend the batch function to allow passing its own allocation function; the current benchmarks would not be calling into direct reclaim today.
Wei noted that we do not have a mechanism today to be able to compare relative coldness of memory to make promotion and demotion decisions. He suggested using a time dimension such that the page being demoted is guaranteed to not have been accessed in the last interval whereas the page being promoted *is* guaranteed to have been accessed in that interval.
----->o-----
Next meeting will be on Thursday, February 26 at 8:30am PST (UTC-8), everybody is welcome: https://meet.google.com/jak-ytdx-hnm
Topics for the next meeting:
- any on-going work for CHMU and a generic hotness interface to leverage it
- RFC v5 of Bharata's patch series for pghot with two modes of operation
- promotion and demotion thrashing detection and back-off mechanisms
- LSF/MM/BPF 2026 topics to propose for discussion on hotness, promotion, and memory tiering overall
- Gregory's testing of reclaim fairness with Bharata's changes and first posting for an RFC
- discuss generalized subsystem for providing bandwidth information independent of the underlying platform, ideally through resctrl, otherwise utilizing bandwidth information will be challenging
+ preferably this bandwidth monitoring is not per NUMA node but rather slow and fast
- determine minimal viable upstream opportunity to optimize for tiering that is extensible for future use cases and optimizations
+ extensible for multiple tiers
+ must be possible to disable with no memory or performance overhead
- update on non-temporal stores enlightenment for memory tiering
- enlightening migrate_pages() for hardware assists and how this work will be charged to userspace, including for memory compaction
Please let me know if you'd like to propose additional topics for discussion, thank you!