Notes from June 18, 2026
Hi everybody,
Here are the notes from the last Linux Memory Hotness and Promotion call that happened on Thursday, June 18. Thanks to everybody who was involved!
These notes are intended to bring people up to speed who could not attend the call as well as keep the conversation going in between meetings.
----->o-----
Bharata updated that he is getting more benchmark numbers for IBS Memory Profiler source with pghot and he submitted an abstract to LPC for a talk on IBS Memory Profiler. The next version of the pghot patch series should be posted in a couple weeks with new benchmark numbers.
----->o-----
Teja Vojjala updated that he will start working with Shivank's patch series for migrate_pages() and collecting some benchmark numbers. He noted improvement on Zen5 especially for benchmarks that are flushing the cache. He was not so sure about the instruction that the patch is using, whether or not this will pollute the cache when doing demotion from top tier memory to lower tier memory.
We're working on non-temporal writes to avoid unintentionally promoting cold pages that appear as hot otherwise. He also noted that comparison with other approaches showing a degradation when using AMD's patch series. I asked him to follow up on Shivank's patch series upstream to have the discussion.
----->o-----
We skipped ahead to discussing non-temporal stores given its overlap with work being done in migrate_pages(). Yiannis had posted his patch series as follow-up to the previous call in the series. He noted that he took one of the existing functions for the pmem era and that it may be possible to do something better in terms of implementation. We see reduced cache pollution with these patches measured by the CXL device. We see a complete elimination of read traffic on the device itself as a result.
Gregory Price provided feedback upstream about the use of the Kconfig in the patch series. That was his only question for the series itself, he thought we should be able to identify if we can do this offload through cpuid; if so, then we could always just use it. Yiannis noted that with this implementation you could see a much higher number of instructions retired which may or may not matter. The bigger benefit here was that it reduces bandwidth. Gregory didn't think of a scenario where we would ever want to turn this off. Jonathan Cameron suggested removing the Kconfig option.
I pressed on the new migrate_mode addition given the long-term precedent of the existing migrate modes. Why do we need a new migrate_mode that specifies whether we want NT stores or not? Wei noted that Google does the same thing because there are otherwise ABI issues which is a bigger problem because the migration callback does not include the reason passed to migrate_pages(). I was surprised that the only way to change migration behavior today was to introduce a new migrate_mode that would be passed into migrate_pages() and that this didn't seem very extensible.
Yiannis will refresh the patch series to not include the Kconfig and sent out an RFC v2.
Wei noted that some Google work was supporting memory errors while using NT stores and that we could discuss that in a future call. This would be a natural extnesion to what Yiannis is working on.
----->o-----
Joshua updated on local testing for tier-aware memcg limits. He ran some benchmarks where DRAM performance was the exact same as CXL performance to identify the impact of the overall approach itself. The overhead was minimized so far to 1-2%. He also did testing on real hardware and got similar results. He will be going through this testing before posting the next update.
----->o-----
The next meeting will be CANCELED due to the Independence Day Holiday in the United States.
Next meeting will be on Thursday, July 16 at 8:30am PDT (UTC-7), everybody is welcome: https://meet.google.com/jak-ytdx-hnm
Topics for the next meeting:
- LPC planning for October 5-7 in Prague
- update on Yiannis's RFC for non-temporal stores enlightenment in migrate_pages()
- extensions to ensure that non-temporal stores can properly handle memory errors
- v7 of Bharata's patch series, including new IBS hotness information and NUMBA2 hint faults
- v5 of Shivank's series for enlightening migrate_pages() for hardware assists and how this work will be charged to userspace, including for memory compaction, and separating the first patches out into a non-RFC series
- v2 of tier-aware memcg limits, including ABI changes to support more than two tiers
- first class support for virtualization based memory tier support, how to leverge memory tiers in the guest
- discuss generalized subsystem for providing bandwidth information independent of the underlying platform, ideally through resctrl, otherwise utilizing bandwidth information will be challenging
+ preferably this bandwidth monitoring is not per NUMA node but rather slow and fast
- later: testing of tier aware memcg limits with Bharata's changes once tier aware memcg limits is stable and further along
Please let me know if you'd like to propose additional topics for discussion, thank you!
Quoted 47 lines · David Rientjes
Hi everybody,
Here are the notes from the last Linux Memory Hotness and Promotion call that happened on Thursday, June 18. Thanks to everybody who was involved!
These notes are intended to bring people up to speed who could not attend the call as well as keep the conversation going in between meetings.
----->o-----
Bharata updated that he is getting more benchmark numbers for IBS Memory Profiler source with pghot and he submitted an abstract to LPC for a talk on IBS Memory Profiler. The next version of the pghot patch series should be posted in a couple weeks with new benchmark numbers.----->o-----
Teja Vojjala updated that he will start working with Shivank's patch series formigrate_pages()and collecting some benchmark numbers. He noted improvement on Zen5 especially for benchmarks that are flushing the cache. He was not so sure about the instruction that the patch is using, whether or not this will pollute the cache when doing demotion from top tier memory to lower tier memory.We're working on non-temporal writes to avoid unintentionally promoting cold pages that appear as hot otherwise. He also noted that comparison with other approaches showing a degradation when using AMD's patch series. I asked him to follow up on Shivank's patch series upstream to have the discussion.
----->o-----
We skipped ahead to discussing non-temporal stores given its overlap with work being done inmigrate_pages(). Yiannis had posted his patch series as follow-up to the previous call in the series. He noted that he took one of the existing functions for the pmem era and that it may be possible to do something better in terms of implementation. We see reduced cache pollution with these patches measured by the CXL device. We see a complete elimination of read traffic on the device itself as a result.Gregory Price provided feedback upstream about the use of the
Kconfigin the patch series. That was his only question for the series itself, he thought we should be able to identify if we can do this offload through cpuid; if so, then we could always just use it. Yiannis noted that with this implementation you could see a much higher number of instructions retired which may or may not matter. The bigger benefit here was that it reduces bandwidth. Gregory didn't think of a scenario where we would ever want to turn this off. Jonathan Cameron suggested removing theKconfigoption.
Quoted 9 lines
I pressed on the new
migrate_modeaddition given the long-term precedent of the existing migrate modes. Why do we need a newmigrate_modethat specifies whether we want NT stores or not? Wei noted that Google does the same thing because there are otherwise ABI issues which is a bigger problem because the migration callback does not include the reason passed tomigrate_pages(). I was surprised that the only way to change migration behavior today was to introduce a newmigrate_modethat would be passed intomigrate_pages()and that this didn't seem very extensible.
struct migrate_ctx {
enum migrate_mode mode;
/* copy method: regular, non-temporal, offload ...? */
/* reason, other info ... can be extended as per requirement */
};int (*migrate_folio)(struct address_space *mapping, struct folio *dst,
struct folio *src, struct migrate_ctx *ctx);
Should we extend the migrate_folio() to pass a context struct instead of just migrate_mode?
Changing this callback is actually not that complex (only 59 instances .migrate_folio =), and this will fix the migrate_folio() once and for all.
I'm working on simplifying the migration code as it's become increasingly complex to add new optimization.
Thanks,
Shivank
Quoted 50 lines
Yiannis will refresh the patch series to not include the
Kconfigand sent out an RFC v2.Wei noted that some Google work was supporting memory errors while using NT stores and that we could discuss that in a future call. This would be a natural extnesion to what Yiannis is working on.
----->o-----
Joshua updated on local testing for tier-aware memcg limits. He ran some benchmarks where DRAM performance was the exact same as CXL performance to identify the impact of the overall approach itself. The overhead was minimized so far to 1-2%. He also did testing on real hardware and got similar results. He will be going through this testing before posting the next update.----->o-----
The next meeting will be CANCELED due to the Independence Day Holiday in the United States.Next meeting will be on Thursday, July 16 at 8:30am PDT (UTC-7), everybody is welcome: https://meet.google.com/jak-ytdx-hnm
Topics for the next meeting:
- LPC planning for October 5-7 in Prague
- update on Yiannis's RFC for non-temporal stores enlightenment inmigrate_pages()
- extensions to ensure that non-temporal stores can properly handle memory errors
- v7 of Bharata's patch series, including new IBS hotness information and NUMBA2 hint faults
- v5 of Shivank's series for enlighteningmigrate_pages()for hardware assists and how this work will be charged to userspace, including for memory compaction, and separating the first patches out into a non-RFC series
- v2 of tier-aware memcg limits, including ABI changes to support more than two tiers
- first class support for virtualization based memory tier support, how to leverge memory tiers in the guest
- discuss generalized subsystem for providing bandwidth information independent of the underlying platform, ideally through resctrl, otherwise utilizing bandwidth information will be challenging
+ preferably this bandwidth monitoring is not per NUMA node but rather slow and fast
- later: testing of tier aware memcg limits with Bharata's changes once tier aware memcg limits is stable and further alongPlease let me know if you'd like to propose additional topics for discussion, thank you!