Notes from May 21, 2026
Hi everybody,
Here are the notes from the last Linux Memory Hotness and Promotion call that happened on Thursday, May 21. Thanks to everybody who was involved!
These notes are intended to bring people up to speed who could not attend the call as well as keep the conversation going in between meetings.
----->o-----
Yiannis updated offline that he is planning on sending the RFC for non-temporal stores in migrate_pages() by tomorrow. Upstream feedback is very appreciated.
We introduced a new addition to the group, Frank van der Linden from Google. He's focused on memory tiering and will be participating in the upstream calls related to it. I gave a brief overview of the topics covered by the meeting and the usual participants.
----->o-----
Joshua updated on his experience at LSF/MM/BPF. The direction for his patch series looks good but will require some ABI changes; some feedback at the conference discussed the eventual need to support 3+ tiers, so extensibility of the interfaces is important.
There was some feedback offered at the conference from Google that memory tiering may not be here for another decade or so. There was a hypothesis that this was related to the DRAM shortage in the industry and there was an idea about managing the tiers separately because there were dollar amount differences in cost.
Myself and Wei corrected this stance from Google: memory tiering is deployed in production and is here to stay. We are strategically very invested in this going forward, so it was unclear where the feedback was being offered from. Wei also noted that we are actively expanding into three tier configurations and working on how to integrate swap into this topology.
Joshua discussed swap as a memory tier. He mentioned how cgroup v1 has a concept of memsw and how it treats swap as part of the memory footprint. Wei said one challenge is how we move memory between these tiers: there are similarities in the demotion path in using the LRU to find cold pages. The different tiers have different performance characteristics, however. In addition, CXL memory tiers can do direct access. The mechanisms and policies will be very important.
Wei suggested that we prioritize discussions on how memory tiers can be used for virtualization purposes and what role the guest plays. We'll add this to the agenda for future discussion. Joshua noted that Gregory has been looking into how the guest can become aware of tiers on the host or whether it's abstracted away, so a very interesting discussion could be had about this area.
----->o-----
Shivank updated on the status of his patch series. He will be breaking out the first few patches from his series and sending them upstream as non-RFC changes. Offloading and DMA will follow. I reiterated that Google will be happy to test and eventually benchmark these patches to support their upstream inclusion.
We discussed leveraging the migration reason within the migrate_pages() call to determine when offloading should be used. I preferred to ensure that we have the flexibility to offload certain types of migration, such as only promotions and demotions, rather than all migrations to make more efficient use of the hardware assist. I thought that heuristics based on either page size or batch size wouldn't allow for this flexibility and hard code behavior that may not be desired.
Shivank also pointed to patch series that are current work in progress in this area (thank you!)[1][2][3][4].
----->o-----
Next meeting will be on Thursday, June 4 at 8:30am PDT (UTC-7), everybody is welcome: https://meet.google.com/jak-ytdx-hnm
Topics for the next meeting:
- v7 of Bharata's patch series, including new IBS hotness information and NUMBA2 hint faults
- v5 of Shivank's series for enlightening migrate_pages() for hardware assists and how this work will be charged to userspace, including for memory compaction, and separating the first patches out into a non-RFC series
- v2 of tier-aware memcg limits, including ABI changes to support more than two tiers
- update on Yiannis's RFC for non-temporal stores enlightenment in migrate_pages()
- first class support for virtualization based memory tier support, how to leverge memory tiers in the guest
- discuss generalized subsystem for providing bandwidth information independent of the underlying platform, ideally through resctrl, otherwise utilizing bandwidth information will be challenging
+ preferably this bandwidth monitoring is not per NUMA node but rather slow and fast
- later: testing of tier aware memcg limits with Bharata's changes once tier aware memcg limits is stable and further along
Please let me know if you'd like to propose additional topics for discussion, thank you!
[1]
https://lore.kernel.org/all/[email protected]
[2]
https://github.com/shivankgarg98/linux/tree/shivank/V5_migrate_pages_optimization
[3]
https://lore.kernel.org/linux-mm/[email protected]/
[4]
https://lore.kernel.org/all/[email protected]
Quoted 25 lines · David Rientjes
Next meeting will be on Thursday, June 4 at 8:30am PDT (UTC-7), everybody is welcome: https://meet.google.com/jak-ytdx-hnm
Topics for the next meeting:
- v7 of Bharata's patch series, including new IBS hotness information and NUMBA2 hint faults
- v5 of Shivank's series for enlighteningmigrate_pages()for hardware assists and how this work will be charged to userspace, including for memory compaction, and separating the first patches out into a non-RFC series
- v2 of tier-aware memcg limits, including ABI changes to support more than two tiers
- update on Yiannis's RFC for non-temporal stores enlightenment inmigrate_pages()
- first class support for virtualization based memory tier support, how to leverge memory tiers in the guest
- discuss generalized subsystem for providing bandwidth information independent of the underlying platform, ideally through resctrl, otherwise utilizing bandwidth information will be challenging
+ preferably this bandwidth monitoring is not per NUMA node but rather slow and fast
- later: testing of tier aware memcg limits with Bharata's changes once tier aware memcg limits is stable and further along
Quoted 2 lines
Please let me know if you'd like to propose additional topics for discussion, thank you!
Apologies for being a bit quiet recently, have had a number of disruptions and have been on parental leave.
I should be back mid-june (so not next meeting, but the following).
I've been investigating LRU inversions in vmscan.c in multi-tier systems [0] and have been testing why i call "Preclaim" - i.e. direct reclaimers helping to reclaim memory from lower nodes instead of swapping directly from higher nodes.
I have a patch set that I think everyone may want to take a look at but haven't published yet. It's especially bad when proactive reclaim is used and lower tier memory is perpetually full.
[0] https://lore.kernel.org/linux-mm/agHnYo_yvyXHBtTJ@gourry-fedora-PF4VCD3F/
~Gregory