Notes from December 4, 2025
Hi everybody,
Here are the notes from the last Linux Memory Hotness and Promotion call that happened on Thursday, December 4. Thanks to everybody who was involved!
These notes are intended to bring people up to speed who could not attend the call as well as keep the conversation going in between meetings.
----->o-----
Bharata updated that he is ready to post v4 of his series with two main
changes: per section indicator for hotness, which reduces the effort required for kmigrated, and then folio_mark_accessed() that shows good results in initial testing. Bharata is also working on getting the right configuration for redis-memtier to ensure we are moving memory back and forth for promotion and demotion.
Discussing the consolidation of tunables, the current plan was to ensure that these live under sysfs. There was discussion about doing this in debugfs first, but the API should be solidified enough by the time of upstream inclusion that this won't be necessary.
----->o-----
I pivoted the discussion to talk about single threaded vs multi threaded promotion. Wei Xu noted that we likely need more than one thread, probably at least one per NUMA node. He said the memory bandwidth contention was so severe that promotion was very reduced. Hardware assist can help, and so can memory bandwidth QoS can help, but another option is also multi threaded promotions. Jonathan Cameron asked if multiple threads were being used for QoS; Wei said that by moving the memory off the low tier that we are shifting the bandwidth elsewhere. The ideal scenario, Wei said, was to ensure that hardware based memory bandwidth QoS would ensure that the promotion threads can make steady progress.
Gregory suggested this may be a transient factor, we've had the discussion before about how aggressive tiering should be to improve overall latency. Moving things as fast as possible may not always be the desired effect, the goal is to converge on stability so you may not maximize bandwidth to balance memory but rather eventually reach an ideal steady state. Wei shared that in some cases this was as limited as <100MB/s for promotion as a result of bandwidth saturation. Gregory asked how realistic of a scenario this actually is because the fact that we have gotten into this situation is already problematic: the only scenario whre this "should" happen is if the DRAM tier bandwidth is already capped and the amount of hot memory exceeds the DRAM tier capacity in which case we would have thrashing. The goal is to ensure this scenario doesn't happen at all.
Jonathan said boosting the migration threads may not be the ultimate solution, but rather reducing the workload that is using all the bandwidth from being scheduled. Wei believed that a single threaded promotion thread could still be a bottleneck. Gregory suggested that 100MB/s promotion may not be too slow, the goal is to eventually get there but not by promoting memory over a very short window -- if we promote as fast as possible, we could find that memory does not remain hot in which case the promotion may not have been justified.
----->o-----
We shifted to discussing about workloads that can be used for experimentation. Gregory suggested we reached critical mass on the topic such that we would really benefit from production data or actual deployments. He took the AI to do this, although admittedly it may take some time.
Wei had been working primarily with memcached although production data was not imminent.
----->o-----
Next meeting will be on Thursday, December 18 at 8:30am PST (UTC-8), everybody is welcome: https://meet.google.com/jak-ytdx-hnm
Topics for the next meeting:
- updates on Bharata's RFC v4 with new benchmarks and consolidation of tunables
- continued discussion on memory overheads used to save the memory hotness state and the list of promotion targets
- workloads to use as the industry standard beyond just memcache, such as redis
- later: Gregory's analysis of more production-like workloads
- discuss generalized subsystem for providing bandwidth information independent of the underlying platform, ideally through resctrl, otherwise utilizing bandwidth information will be challenging
+ preferably this bandwidth monitoring is not per NUMA node but rather slow and fast
- similarly, discuss generalized subsystem for providing memory hotness information
- determine minimal viable upstream opportunity to optimize for tiering that is extensible for future use cases and optimizations
- update on non-temporal stores enlightenment for memory tiering
- enlightening migrate_pages() for hardware assists and how this work will be charged to userspace, including for memory compaction
Please let me know if you'd like to propose additional topics for discussion, thank you!