Kernel daemon for detecting and promoting hot pages
Subsystems (8)
cover Kernel daemon for detecting and promoting hot pages
Hi,
This is an attempt towards having a single subsystem that accumulates hot page information from lower memory tiers and does hot page promotion.
At the heart of this subsystem is a kernel daemon named kpromoted that does the following:
1. Exposes an API that other subsystems which detect/generate memory access information can use to inform the daemon about memory accesses from lower memory tiers.
2. Maintains the list of hot pages and attempts to promote them to toptiers.
Currently I have added AMD IBS driver as one source that provides page access information as an example. This driver feeds info to kpromoted in this RFC patchset. More sources were discussed in a similar context here at [1].
This is just an early attempt to check what it takes to maintain a single source of page hotness info and also separate hot page detection mechanisms from the promotion mechanism. There are too many open ends right now and I have listed a few of them below.
- The API that is provided to register memory access expects the PFN, NID and time of access at the minimum. This is described more in patch 2/4. This API currently can be called only from contexts that allow sleeping and hence this rules out using it from PTE scanning paths. The API needs to be more flexible with respect to this.
- Some sources like PTE A bit scanning can't provide the precise time of access or the NID that is accessing the page. The latter has been an open problem to which I haven't come across a good and acceptable solution.
- The way the hot page information is maintained is pretty primitive right now. Ideally we would like to store hotness info in such a way that it should be easily possible to lookup say N most hot pages.
- If PTE A bit scanners are considered as hotness sources, we will be bombarded with accesses. Do we want to accomodate all those accesses or just go with hotness info for fixed number of pages (possibly as a ratio of lower tier memory capacity)?
- Undoubtedly the mechanism to classify a page as hot and subsequent promotion needs to be more sophisticated than what I have right now.
This is just an early RFC posted now to ignite some discussion in the context of LSFMM [2].
I am also working with Raghu to integrate his kmmdscan [3] as the hotness source and use kpromoted for migration.
Also, I had posted the IBS driver ealier as an alternative to hint faults based NUMA Balancing [4]. However here I am using it as generic page hotness source.
[1] https://lore.kernel.org/linux-mm/[email protected]/
[2] https://lore.kernel.org/linux-mm/[email protected]/
[3] https://lore.kernel.org/all/[email protected]/
[3] https://lore.kernel.org/lkml/[email protected]/
Regards,
Bharata.
Bharata B Rao (4):
mm: migrate: Allow misplaced migration without VMA too
mm: kpromoted: Hot page info collection and promotion daemon
x86: ibs: In-kernel IBS driver for memory access profiling
x86: ibs: Enable IBS profiling for memory accesses
arch/x86/events/amd/ibs.c | 11 + arch/x86/include/asm/entry-common.h | 3 + arch/x86/include/asm/hardirq.h | 2 + arch/x86/include/asm/ibs.h | 9 + arch/x86/include/asm/msr-index.h | 16 ++ arch/x86/mm/Makefile | 3 +- arch/x86/mm/ibs.c | 344 ++++++++++++++++++++++++++++ include/linux/kpromoted.h | 54 +++++ include/linux/mmzone.h | 4 + include/linux/vm_event_item.h | 30 +++ mm/Kconfig | 7 + mm/Makefile | 1 + mm/kpromoted.c | 305 ++++++++++++++++++++++++ mm/migrate.c | 5 +- mm/mm_init.c | 10 + mm/vmstat.c | 30 +++ 16 files changed, 831 insertions(+), 3 deletions(-) create mode 100644 arch/x86/include/asm/ibs.h create mode 100644 arch/x86/mm/ibs.c create mode 100644 include/linux/kpromoted.h create mode 100644 mm/kpromoted.c
Signature · 1 line
2.34.1
RFC PATCH 1/4 mm: migrate: Allow misplaced migration without VMA too
migrate_misplaced_folio_prepare() can be called from a context where VMA isn't available. Allow the migration to work from such contexts too.
--- mm/migrate.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
mm/migrate.c +3 -2
@@ -2639,7 +2639,8 @@ static struct folio *alloc_misplaced_dst_folio(struct folio *src,/* * Prepare for calling migrate_misplaced_folio() by isolating the folio if -* permitted. Must be called with the PTL still held. +* permitted. Must be called with the PTL still held if called with a non-NULL +* vma. */ int migrate_misplaced_folio_prepare (struct folio *folio,struct vm_area_struct *vma,int node)
@@ -2656,7 +2657,7 @@ int migrate_misplaced_folio_prepare(struct folio *folio,* See folio_likely_mapped_shared() on possible imprecision * when we cannot easily detect if a folio is shared. */ -if ((vma->vm_flags &VM_EXEC ) && +if (vma && (vma->vm_flags &VM_EXEC ) &&folio_likely_mapped_shared (folio))return -EACCES ;
Signature · 1 line
2.34.1
Re: [RFC PATCH 1/4] mm: migrate: Allow misplaced migration without VMA too
Quoted 3 lines · Bharata B Rao
migrate_misplaced_folio_prepare()can be called from a context where VMA isn't available. Allow the migration to work from such contexts too.
I was initially confused about "can be called", because it can't
Consider phrasing it as "We want to make use of alloc_misplaced_dst_folio() in context where we don't have VMA information available. To prepare for that ..."
Quoted 29 lines
Signed-off-by Bharata B Rao <[email protected]>--- mm/migrate.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)mm/migrate.c +3
-2@@ -2639,7 +2639,8 @@ static struct folio *alloc_misplaced_dst_folio(struct folio *src,/* * Prepare for calling migrate_misplaced_folio() by isolating the folio if -* permitted. Must be called with the PTL still held. +* permitted. Must be called with the PTL still held if called with a non-NULL +* vma. */ int migrate_misplaced_folio_prepare (struct folio *folio,struct vm_area_struct *vma,int node)@@ -2656,7 +2657,7 @@ int migrate_misplaced_folio_prepare(struct folio *folio,* See folio_likely_mapped_shared() on possible imprecision * when we cannot easily detect if a folio is shared. */ -if ((vma->vm_flags &VM_EXEC ) && +if (vma && (vma->vm_flags &VM_EXEC ) &&folio_likely_mapped_shared (folio))return -EACCES ;
Signature · 3 lines
Cheers, David / dhildenb
Re: [RFC PATCH 1/4] mm: migrate: Allow misplaced migration without VMA too
Quoted 8 lines · David Hildenbrand
Quoted 3 lines · Bharata B Rao
migrate_misplaced_folio_prepare()can be called from a context where VMA isn't available. Allow the migration to work from such contexts too.I was initially confused about "can be called", because it can't
Consider phrasing it as "We want to make use of
alloc_misplaced_dst_folio()in context where we don't have VMA information available. To prepare for that ..."
Yes, that would be the right wording.
Thanks,
Bharata.
Gregory Price
reviewerRe: [RFC PATCH 1/4] mm: migrate: Allow misplaced migration without VMA too
In reply to Bharata B RaoQuoted 5 lines · Bharata B Rao
migrate_misplaced_folio_prepare()can be called from a context where VMA isn't available. Allow the migration to work from such contexts too.Signed-off-by Bharata B Rao <[email protected]>
I have a similar patch in the unmapped pagecache RFC
we may also need this:
https://lore.kernel.org/linux-mm/[email protected]/
May be worth just pulling these ahead to avoid conflict.
~Gregory
Re: [RFC PATCH 1/4] mm: migrate: Allow misplaced migration without VMA too
Quoted 5 lines · Gregory Price
Quoted 4 lines · Bharata B Rao
migrate_misplaced_folio_prepare()can be called from a context where VMA isn't available. Allow the migration to work from such contexts too.
Quoted 8 lines
Quoted 1 line
Signed-off-by Bharata B Rao <[email protected]>I have a similar patch in the unmapped pagecache RFC
we may also need this:
https://lore.kernel.org/linux-mm/[email protected]/May be worth just pulling these ahead to avoid conflict.
Or not putting them in at all because this whole thing is a magnificent waste of time?
Gregory Price
reviewerRe: [RFC PATCH 1/4] mm: migrate: Allow misplaced migration without VMA too
Quoted 8 lines · Matthew Wilcox
Quoted 4 lines · Gregory Price
we may also need this:
https://lore.kernel.org/linux-mm/[email protected]/May be worth just pulling these ahead to avoid conflict.
Or not putting them in at all because this whole thing is a magnificent waste of time?
Divorced from the tiering mechanisms, is making misplaced migration able to migrate unmapped pages not generally useful?
~Gregory
Re: [RFC PATCH 1/4] mm: migrate: Allow misplaced migration without VMA too
Quoted 5 lines · Gregory Price
Quoted 4 lines · Matthew Wilcox
Quoted 3 lines · Gregory Price
we may also need this:
https://lore.kernel.org/linux-mm/[email protected]/
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
May be worth just pulling these ahead to avoid conflict.
Or not putting them in at all because this whole thing is a magnificent waste of time?
Divorced from the tiering mechanisms, is making misplaced migration able to migrate unmapped pages not generally useful?
The only thing I can think of is if you have a process or set of processes on node A calling read() and the file is cached on node B. But in order to decide if the page is on the wrong node, you'd need to track a lot of information about which nodes the page is being accessed from. Which is probably why we've never bothered to do it.
This is not a large patch for you to carry as part of your patchset. There's nothing intrinsically wrong with it; it just has no users in mainline and no real prospect of any being added soon.
Gregory Price
reviewerRe: [RFC PATCH 1/4] mm: migrate: Allow misplaced migration without VMA too
Quoted 10 lines · Matthew Wilcox
Quoted 2 lines · Gregory Price
Divorced from the tiering mechanisms, is making misplaced migration able to migrate unmapped pages not generally useful?
The only thing I can think of is if you have a process or set of processes on node A calling
read()and the file is cached on node B. But in order to decide if the page is on the wrong node, you'd need to track a lot of information about which nodes the page is being accessed from. Which is probably why we've never bothered to do it.
Quoted 3 lines
This is not a large patch for you to carry as part of your patchset. There's nothing intrinsically wrong with it; it just has no users in mainline and no real prospect of any being added soon.
That's fair, I'm just tracking 3-4 different RFCs that are going to butt up against this, so wanted to assess whether getting the patches out ahead would save some strife.
~Gregory
Re: [RFC PATCH 1/4] mm: migrate: Allow misplaced migration without VMA too
In reply to Bharata B RaoQuoted 24 lines · Bharata B Rao
migrate_misplaced_folio_prepare()can be called from a context where VMA isn't available. Allow the migration to work from such contexts too.Signed-off-by Bharata B Rao <[email protected]>--- mm/migrate.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)mm/migrate.c +2
-1@@ -2639,7 +2639,8 @@ static struct folio *alloc_misplaced_dst_folio(struct folio *src,/* * Prepare for calling migrate_misplaced_folio() by isolating the folio if -* permitted. Must be called with the PTL still held. +* permitted. Must be called with the PTL still held if called with a non-NULL +* vma. */ int migrate_misplaced_folio_prepare (struct folio *folio,struct vm_area_struct *vma,int node)
mm/migrate.c int migrate_misplaced_folio_prepare(struct folio *folio, · line 2663
Quoted 8 lines
mm/migrate.c +1
-1@@ -2656,7 +2657,7 @@ int migrate_misplaced_folio_prepare(struct folio *folio,* See folio_likely_mapped_shared() on possible imprecision * when we cannot easily detect if a folio is shared. */ -if ((vma->vm_flags &VM_EXEC ) && +if (vma && (vma->vm_flags &VM_EXEC ) &&folio_likely_mapped_shared (folio))return -EACCES ;
In the worst case, the absence of the vma would mean that we try to isolate and migrate a shared folio with executable pages. Are those a key target for the hot page migration?
Balbir
Re: [RFC PATCH 1/4] mm: migrate: Allow misplaced migration without VMA too
Quoted 29 lines · Balbir Singh
Quoted 28 lines · Bharata B Rao
migrate_misplaced_folio_prepare()can be called from a context where VMA isn't available. Allow the migration to work from such contexts too.Signed-off-by Bharata B Rao <[email protected]>--- mm/migrate.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)mm/migrate.c +2
-2@@ -2639,7 +2639,8 @@ static struct folio *alloc_misplaced_dst_folio(struct folio *src,/* * Prepare for calling migrate_misplaced_folio() by isolating the folio if -* permitted. Must be called with the PTL still held. +* permitted. Must be called with the PTL still held if called with a non-NULL +* vma. */ int migrate_misplaced_folio_prepare (struct folio *folio,struct vm_area_struct *vma,int node)@@ -2656,7 +2657,7 @@ int migrate_misplaced_folio_prepare(struct folio *folio,* See folio_likely_mapped_shared() on possible imprecision * when we cannot easily detect if a folio is shared. */ - if ((vma->vm_flags & VM_EXEC) &&
Quoted 8 lines
mm/migrate.c int migrate_misplaced_folio_prepare(struct folio *folio, · line 2663
Quoted 4 lines
mm/migrate.c +1
-0+if (vma && (vma->vm_flags &VM_EXEC ) &&folio_likely_mapped_shared (folio))return -EACCES ;In the worst case, the absence of the vma would mean that we try to isolate and migrate a shared folio with executable pages. Are those a key target for the hot page migration?
I don't think they are a key target for hot page migration, but if shared executable pages (like shared library pages) have ended up in lower tier, it doesn't hurt to get them promoted to top tier, I would think.
Regards,
Bharata.
RFC PATCH 2/4 mm: kpromoted: Hot page info collection and promotion daemon
In reply to Bharata B Raokpromoted is a kernel daemon that accumulates hot page info from different sources and tries to promote pages from slow tiers to top tiers. One instance of this thread runs on each node that has CPUs.
Subsystems that generate hot page access info can report that to kpromoted via this API:
int kpromoted_record_access(u64 pfn, int nid, int src,
unsigned long time)
@pfn: The PFN of the memory accessed
@nid: The accessing NUMA node ID
@src: The temperature source (subsystem) that generated the
access info
@time: The access time in jiffies
Some temperature sources may not provide the nid from which the page was accessed. This is true for sources that use page table scanning for PTE Accessed bit. Currently the toptier node to which such pages should be promoted to is hard coded.
Also, the access time provided some sources may at best be considered approximate. This is especially true for hot pages detected by PTE A bit scanning.
kpromoted currently maintains the hot PFN records in hash lists hashed by PFN value. Each record stores the following info:
struct page_hotness_info {
unsigned long pfn;
/* Time when this record was updated last */
unsigned long last_update;
/*
* Number of times this page was accessed in the
* current window
*/
int frequency;
/* Most recent access time */
unsigned long recency;
/* Most recent access from this node */
int hot_node;
struct hlist_node hnode;
};The way in which a page is categorized as hot enough to be promoted is pretty primitive now.
--- include/linux/kpromoted.h | 54 ++++++ include/linux/mmzone.h | 4 + include/linux/vm_event_item.h | 13 ++ mm/Kconfig | 7 + mm/Makefile | 1 + mm/kpromoted.c | 305 ++++++++++++++++++++++++++++++++++ mm/mm_init.c | 10 ++ mm/vmstat.c | 13 ++ 8 files changed, 407 insertions(+) create mode 100644 include/linux/kpromoted.h create mode 100644 mm/kpromoted.c
include/linux/kpromoted.h new file +54 -0
@@ -0,0 +1,54 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_KPROMOTED_H +#define _LINUX_KPROMOTED_H + +#include <linux/types.h> +#include <linux/init.h> +#include <linux/workqueue_types.h> + +/* Page hotness temperature sources */ +enum kpromoted_src { +KPROMOTED_HW_HINTS , +KPROMOTED_PGTABLE_SCAN , +}; + +#ifdef CONFIG_KPROMOTED + +#define KPROMOTED_FREQ_WINDOW (5 * MSEC_PER_SEC) + +/* 2 accesses within a window will make the page a promotion candidate */ +#define KPRMOTED_FREQ_THRESHOLD 2 + +#define KPROMOTED_HASH_ORDER 16 + +struct page_hotness_info { +unsigned long pfn; + +/* Time when this record was updated last */ +unsigned long last_update; + +/* +* Number of times this page was accessed in the +* current window +*/ +int frequency; + +/* Most recent access time */ +unsigned long recency; + +/* Most recent access from this node */ +int hot_node; +struct hlist_node hnode; +}; + +#define KPROMOTE_DELAY MSEC_PER_SEC + +int kpromoted_record_access (u64 pfn,int nid,int src,unsigned long now); +#else +static inline int kpromoted_record_access (u64 pfn,int nid,int src, +unsigned long now) +{ +return 0 ; +} +#endif /* CONFIG_KPROMOTED */ +#endif /* _LINUX_KPROMOTED_H */
include/linux/mmzone.h +4 -0
@@ -1459,6 +1459,10 @@ typedef struct pglist_data {#ifdef CONFIG_MEMORY_FAILURE struct memory_failure_stats mf_stats;#endif +#ifdef CONFIG_KPROMOTED +struct task_struct *kpromoted; +wait_queue_head_t kpromoted_wait; +#endif } pg_data_t;#define node_present_pages (nid) (NODE_DATA(nid)->node_present_pages)
include/linux/vm_event_item.h +13 -0
@@ -182,6 +182,19 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, KSTACK_REST,#endif #endif /* CONFIG_DEBUG_STACK_USAGE */ +KPROMOTED_RECORDED_ACCESSES , +KPROMOTED_RECORD_HWHINTS , +KPROMOTED_RECORD_PGTSCANS , +KPROMOTED_RECORD_TOPTIER , +KPROMOTED_RECORD_ADDED , +KPROMOTED_RECORD_EXISTS , +KPROMOTED_MIG_RIGHT_NODE , +KPROMOTED_MIG_NON_LRU , +KPROMOTED_MIG_COLD_OLD , +KPROMOTED_MIG_COLD_NOT_ACCESSED , +KPROMOTED_MIG_CANDIDATE , +KPROMOTED_MIG_PROMOTED , +KPROMOTED_MIG_DROPPED ,NR_VM_EVENT_ITEMS };
mm/Kconfig +7 -0
@@ -1358,6 +1358,13 @@ config PT_RECLAIM Note: now only empty user PTE page table pages will be reclaimed. +config KPROMOTED + bool "Kernel hot page promotion daemon" + def_bool y + depends on NUMA && MIGRATION && MMU + help + Promote hot pages from lower tier to top tier by using the + memory access information provided by various sources. source "mm/damon/Kconfig"
mm/Makefile +1 -0
@@ -147,3 +147,4 @@ obj-$(CONFIG_SHRINKER_DEBUG) += shrinker_debug.o obj-$(CONFIG_EXECMEM) += execmem.o obj-$(CONFIG_TMPFS_QUOTA) += shmem_quota.o obj-$(CONFIG_PT_RECLAIM) += pt_reclaim.o +obj-$(CONFIG_KPROMOTED) += kpromoted.o
mm/kpromoted.c new file +305 -0
305 lines not on this page · raw message
mm/mm_init.c +10 -0
@@ -1362,6 +1362,15 @@ static void pgdat_init_kcompactd(struct pglist_data *pgdat)static void pgdat_init_kcompactd (struct pglist_data *pgdat) {}#endif +#ifdef CONFIG_KPROMOTED +static void pgdat_init_kpromoted (struct pglist_data *pgdat) +{ +init_waitqueue_head (&pgdat->kpromoted_wait); +} +#else +static void pgdat_init_kpromoted (struct pglist_data *pgdat) {} +#endif +static void __meminit pgdat_init_internals (struct pglist_data *pgdat) {int i;
@@ -1371,6 +1380,7 @@ static void __meminit pgdat_init_internals(struct pglist_data *pgdat)pgdat_init_split_queue (pgdat);pgdat_init_kcompactd (pgdat); +pgdat_init_kpromoted (pgdat);init_waitqueue_head (&pgdat->kswapd_wait);init_waitqueue_head (&pgdat->pfmemalloc_wait);
mm/vmstat.c +13 -0
@@ -1466,6 +1466,19 @@ const char * const vmstat_text[] = { "kstack_rest",#endif #endif +"kpromoted_recorded_accesses" , +"kpromoted_recorded_hwhints" , +"kpromoted_recorded_pgtscans" , +"kpromoted_record_toptier" , +"kpromoted_record_added" , +"kpromoted_record_exists" , +"kpromoted_mig_right_node" , +"kpromoted_mig_non_lru" , +"kpromoted_mig_cold_old" , +"kpromoted_mig_cold_not_accessed" , +"kpromoted_mig_candidate" , +"kpromoted_mig_promoted" , + "kpromoted_mig_dropped", #endif /* CONFIG_VM_EVENT_COUNTERS || CONFIG_MEMCG */ };#endif /* CONFIG_PROC_FS || CONFIG_SYSFS || CONFIG_NUMA || CONFIG_MEMCG */
Signature · 1 line
2.34.1
Re: [RFC PATCH 2/4] mm: kpromoted: Hot page info collection and promotion daemon
Quoted 30 lines · Bharata B Rao
mm/kpromoted.c new file +30
-0+static void kpromoted_migrate (pg_data_t *pgdat) +{ +int nid = pgdat->node_id; +struct page_hotness_info *phi; +struct hlist_node *tmp; +int nr_bkts =HASH_SIZE (page_hotness_hash); +int bkt; + +for (bkt =0 ; bkt < nr_bkts; bkt++) { +mutex_lock (&page_hotness_lock[bkt]); +hlist_for_each_entry_safe (phi, tmp, &page_hotness_hash[bkt], hnode) { +if (phi->hot_node != nid) +continue ; + +if (page_should_be_promoted (phi)) { +count_vm_event (KPROMOTED_MIG_CANDIDATE ); +if (!kpromote_page (phi)) { +count_vm_event (KPROMOTED_MIG_PROMOTED ); +hlist_del_init (&phi->hnode); +kfree (phi); + } + }else { +/* +* Not a suitable page or cold page, stop tracking it. +* TODO: Identify cold pages and drive demotion? +*/ +count_vm_event (KPROMOTED_MIG_DROPPED ); +hlist_del_init (&phi->hnode); +kfree (phi); + }
mm/kpromoted.c line 119
Quoted 10 lines
mm/kpromoted.c new file +10
-0+ } +mutex_unlock (&page_hotness_lock[bkt]); + } +} + +static struct page_hotness_info *__kpromoted_lookup (unsigned long pfn,int bkt) +{ +struct page_hotness_info *phi; + +hlist_for_each_entry (phi, &page_hotness_hash[bkt], hnode) {
Should this be hlist_for_each_entry_safe(), given that kpromoted_migrate() may be running concurrently?
Mike
Quoted 5 lines
mm/kpromoted.c new file +5
-0+if (phi->pfn == pfn) +return phi; + } +return NULL ; +}
Re: [RFC PATCH 2/4] mm: kpromoted: Hot page info collection and promotion daemon
Quoted 40 lines · Mike Day
Quoted 37 lines · Bharata B Rao
mm/kpromoted.c new file +11
-0+static void kpromoted_migrate (pg_data_t *pgdat) +{ + int nid = pgdat->node_id; + struct page_hotness_info *phi; + struct hlist_node *tmp; + int nr_bkts = HASH_SIZE(page_hotness_hash); + int bkt; + + for (bkt = 0; bkt < nr_bkts; bkt++) { + mutex_lock(&page_hotness_lock[bkt]); + hlist_for_each_entry_safe(phi, tmp, &page_hotness_hash[bkt],hnode) {
+ if (phi->hot_node!= nid)
+ continue;mm/kpromoted.c new file +23
-0+ + if (page_should_be_promoted(phi)) { + count_vm_event(KPROMOTED_MIG_CANDIDATE); + if (!kpromote_page(phi)) { + count_vm_event(KPROMOTED_MIG_PROMOTED); + hlist_del_init(&phi->hnode); + kfree(phi); + } + } else { + /* + * Not a suitable page or cold page, stop tracking it. + * TODO: Identify cold pages and drive demotion? + */ + count_vm_event(KPROMOTED_MIG_DROPPED); + hlist_del_init(&phi->hnode); + kfree(phi); + } + } + mutex_unlock(&page_hotness_lock[bkt]); + } +} + +static struct page_hotness_info *__kpromoted_lookup(unsigned long
Quoted 8 lines
mm/kpromoted.c line 119
Quoted 5 lines
pfn, int bkt)
mm/kpromoted.c new file +4
-0+{ + struct page_hotness_info *phi; + + hlist_for_each_entry(phi, &page_hotness_hash[bkt], hnode) {Should this be
hlist_for_each_entry_safe(), given thatkpromoted_migrate()may be running concurrently?
I don't think so because the migration path can't walk the list concurrently as the lists are protected by mutex.
Regards,
Bharata.
kernel test robot
build botRe: [RFC PATCH 2/4] mm: kpromoted: Hot page info collection and promotion daemon
In reply to Bharata B Raokernel test robot noticed the following build warnings:
Hi Bharata,
[This is a private test report for your RFC patch.] kernel test robot noticed the following build warnings:
[auto build test WARNING on tip/x86/core]
[also build test WARNING on tip/x86/mm linus/master v6.14-rc5] [cannot apply to akpm-mm/mm-everything next-20250306]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Bharata-B-Rao/mm-migrate-Allow-misplaced-migration-without-VMA-too/20250306-134949 base: tip/x86/core patch link: https://lore.kernel.org/r/20250306054532.221138-3-bharata%40amd.com patch subject: [RFC PATCH 2/4] mm: kpromoted: Hot page info collection and promotion daemon config: s390-randconfig-001-20250307 (https://download.01.org/0day-ci/archive/20250307/[email protected]/config) compiler: s390-linux-gcc (GCC) 14.2.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250307/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All warnings (new ones prefixed by >>):
mm/kpromoted.c: In function 'kpromoted_record_access':
>> mm/kpromoted.c:158:13: warning: variable 'ret' set but not used [-Wunused-but-set-variable]
158 | int ret, bkt;
| ^~~vim +/ret +158 mm/kpromoted.c
147
148 /*
149 * Called by subsystems that generate page hotness/access information.
150 *
151 * Records the memory access info for futher action by kpromoted.
152 */
153 int kpromoted_record_access(u64 pfn, int nid, int src, unsigned long now)
154 {
155 struct page_hotness_info *phi;
156 struct page *page;
157 struct folio *folio;
> 158 int ret, bkt;
159
160 count_vm_event(KPROMOTED_RECORDED_ACCESSES);
161
162 switch (src) {
163 case KPROMOTED_HW_HINTS:
164 count_vm_event(KPROMOTED_RECORD_HWHINTS);
165 break;
166 case KPROMOTED_PGTABLE_SCAN:
167 count_vm_event(KPROMOTED_RECORD_PGTSCANS);
168 break;
169 default:
170 break;
171 }
172
173 /*
174 * Record only accesses from lower tiers.
175 * Assuming node having CPUs as toptier for now.
176 */
177 if (node_is_toptier(pfn_to_nid(pfn))) {
178 count_vm_event(KPROMOTED_RECORD_TOPTIER);
179 return 0;
180 }
181
182 page = pfn_to_online_page(pfn);
183 if (!page || is_zone_device_page(page))
184 return 0;
185
186 folio = page_folio(page);
187 if (!folio_test_lru(folio))
188 return 0;
189
190 bkt = hash_min(pfn, KPROMOTED_HASH_ORDER);
191 mutex_lock(&page_hotness_lock[bkt]);
192 phi = kpromoted_lookup(pfn, bkt, now);
193 if (!phi) {
194 ret = PTR_ERR(phi);
195 goto out;
196 }
197
198 if ((phi->last_update - now) > msecs_to_jiffies(KPROMOTED_FREQ_WINDOW)) {
199 /* New window */
200 phi->frequency = 1; /* TODO: Factor in the history */
201 phi->last_update = now;
202 } else {
203 phi->frequency++;
204 }
205 phi->recency = now;
206
207 /*
208 * TODOs:
209 * 1. Source nid is hard-coded for some temperature sources
210 * 2. Take action if hot_node changes - may be a shared page?
211 * 3. Maintain node info for every access within the window?
212 */
213 phi->hot_node = (nid == NUMA_NO_NODE) ? 1 : nid;
214 mutex_unlock(&page_hotness_lock[bkt]);
215 out:
216 return 0;
217 }
218
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Re: [RFC PATCH 2/4] mm: kpromoted: Hot page info collection and promotion daemon
In reply to Bharata B RaoQuoted 8 lines · Bharata B Rao
mm/kpromoted.c new file +8
-0+static int page_should_be_promoted (struct page_hotness_info *phi) +{ +struct page *page =pfn_to_online_page (phi->pfn); +unsigned long now = jiffies; +struct folio *folio; + +if (!page ||is_zone_device_page (page)) +return false;
mm/kpromoted.c line 56
Quoted 10 lines
mm/kpromoted.c new file +10
-0+ + folio =page_folio (page); +if (!folio_test_lru (folio)) { +count_vm_event (KPROMOTED_MIG_NON_LRU ); +return false; + } +if (folio_nid (folio) == phi->hot_node) { +count_vm_event (KPROMOTED_MIG_RIGHT_NODE ); +return false; + }
How about using the LRU age itself:
if (folio_test_active())
return true;
Quoted 14 lines
mm/kpromoted.c new file +14
-0+ +/* If the page was hot a while ago, don't promote */ +if ((now - phi->last_update) >2 *msecs_to_jiffies (KPROMOTED_FREQ_WINDOW )) { +count_vm_event (KPROMOTED_MIG_COLD_OLD ); +return false; + } + +/* If the page hasn't been accessed enough number of times, don't promote */ +if (phi->frequency <KPRMOTED_FREQ_THRESHOLD ) { +count_vm_event (KPROMOTED_MIG_COLD_NOT_ACCESSED ); +return false; + } +return true; +}
...
mm/kpromoted.c line 242
Quoted 10 lines
mm/kpromoted.c new file +10
-0+static int kpromoted (void *p) +{ +pg_data_t *pgdat = (pg_data_t *)p; +struct task_struct *tsk = current; +long timeout =msecs_to_jiffies (KPROMOTE_DELAY ); + +const struct cpumask *cpumask =cpumask_of_node (pgdat->node_id); + +if (!cpumask_empty (cpumask)) +set_cpus_allowed_ptr (tsk, cpumask);
Explicit cpumasks are not needed if you use kthread_create_on_node().
Quoted 23 lines
mm/kpromoted.c new file +23
-0+ +while (!kthread_should_stop ()) { +wait_event_timeout (pgdat->kpromoted_wait, +kpromoted_work_requested (pgdat), timeout); +kpromoted_do_work (pgdat); + } +return 0 ; +} + +static void kpromoted_run (int nid) +{ +pg_data_t *pgdat =NODE_DATA (nid); + +if (pgdat->kpromoted) +return ; + + pgdat->kpromoted =kthread_run (kpromoted, pgdat,"kpromoted%d" , nid); +if (IS_ERR (pgdat->kpromoted)) { +pr_err ("Failed to start kpromoted on node %d\n" , nid); + pgdat->kpromoted =NULL ; + } +} +
Re: [RFC PATCH 2/4] mm: kpromoted: Hot page info collection and promotion daemon
Quoted 14 lines · Davidlohr Bueso
Quoted 12 lines · Bharata B Rao
mm/kpromoted.c new file +12
-0+static int page_should_be_promoted (struct page_hotness_info *phi) +{ + struct page *page = pfn_to_online_page(phi->pfn); + unsigned long now = jiffies; + struct folio *folio; + + if (!page || is_zone_device_page(page)) + return false; + + folio = page_folio(page); + if (!folio_test_lru(folio)) { + count_vm_event(KPROMOTED_MIG_NON_LRU);
Quoted 8 lines
Quoted 6 lines
+ return false;
+ }
+ if (folio_nid(folio)==phi->hot_node) { +count_vm_event(KPROMOTED_MIG_RIGHT_NODE); + return false;
+ }How about using the LRU age itself:
Sounds like a good check for page hotness.
Quoted 3 lines
if (
folio_test_active())
return true;
But the numbers I obtained with this check added, didn't really hit this condition all that much. I was running a multi-threaded application that allocates enough memory such that the allocation spills over from DRAM node to the CXL node. Threads keep touching the memory pages in random order.
kpromoted_recorded_accesses 960620 /* Number of recorded accesses */kpromoted_recorded_hwhints 960620 /* Nr accesses via HW hints, IBS in this case */ kpromoted_recorded_pgtscans 0kpromoted_record_toptier 638006 /* Nr toptier accesses */kpromoted_record_added 321234 /* Nr (CXL) accesses that are tracked */kpromoted_record_exists 1380kpromoted_mig_right_node 0kpromoted_mig_non_lru 226kpromoted_mig_lru_active 47 /* Number of accesses considered for promotion as determined by folio_test_active() check */ kpromoted_mig_cold_old 0kpromoted_mig_cold_not_accessed 1373kpromoted_mig_candidate 319635kpromoted_mig_promoted 319635kpromoted_mig_dropped 1599
Need to check why is this the case.
Quoted 20 lines
Quoted 17 lines
+
+ /* If the page was hot a while ago, don't promote */
+ if ((now -phi->last_update) > 2 *msecs_to_jiffies(KPROMOTED_FREQ_WINDOW)) {
+count_vm_event(KPROMOTED_MIG_COLD_OLD);
+ return false;
+ }
+
+ /* If the page hasn't been accessed enough number of times, don't promote */
+ if (phi->frequency<KPRMOTED_FREQ_THRESHOLD) {
+count_vm_event(KPROMOTED_MIG_COLD_NOT_ACCESSED);
+ return false;
+ }
+ return true;
+}...
Quoted 12 lines
mm/kpromoted.c line 242
Quoted 10 lines
mm/kpromoted.c new file +10
-0+static int kpromoted (void *p) +{ + pg_data_t *pgdat = (pg_data_t *)p; + struct task_struct *tsk = current; + long timeout = msecs_to_jiffies(KPROMOTE_DELAY); + + const struct cpumask *cpumask = cpumask_of_node(pgdat->node_id); + + if (!cpumask_empty(cpumask)) + set_cpus_allowed_ptr(tsk, cpumask);Explicit cpumasks are not needed if you use
kthread_create_on_node().
Thanks, will incorporate.
Regards,
Bharata.
Gregory Price
reviewerRe: [RFC PATCH 2/4] mm: kpromoted: Hot page info collection and promotion daemon
Quoted 27 lines · Bharata B Rao
Quoted 14 lines · Davidlohr Bueso
Quoted 12 lines · Bharata B Rao
mm/kpromoted.c new file +12
-0+static int page_should_be_promoted (struct page_hotness_info *phi) +{ + struct page *page = pfn_to_online_page(phi->pfn); + unsigned long now = jiffies; + struct folio *folio; + + if (!page || is_zone_device_page(page)) + return false; + + folio = page_folio(page); + if (!folio_test_lru(folio)) { + count_vm_event(KPROMOTED_MIG_NON_LRU);Quoted 8 lines
Quoted 6 lines
+ return false;
+ }
+ if (folio_nid(folio)==phi->hot_node) { +count_vm_event(KPROMOTED_MIG_RIGHT_NODE); + return false;
+ }How about using the LRU age itself:
Sounds like a good check for page hotness.
Quoted 1 line
Quoted 8 lines
Quoted 2 lines
if (
folio_test_active())
return true;But the numbers I obtained with this check added, didn't really hit this condition all that much. I was running a multi-threaded application that allocates enough memory such that the allocation spills over from DRAM node to the CXL node. Threads keep touching the memory pages in random order.
Is demotion enabled by any chance?
i.e. are you sure it's actually allocating from CXL and not demoting cold stuff to CXL?
Quoted 56 lines
kpromoted_recorded_accesses960620 /* Number of recorded accesses */kpromoted_recorded_hwhints960620 /* Nr accesses via HW hints, IBS in this case */kpromoted_recorded_pgtscans0kpromoted_record_toptier638006 /* Nr toptier accesses */kpromoted_record_added321234 /* Nr (CXL) accesses that are tracked */kpromoted_record_exists1380kpromoted_mig_right_node0kpromoted_mig_non_lru226kpromoted_mig_lru_active47 /* Number of accesses considered for promotion as determined byfolio_test_active()check */kpromoted_mig_cold_old0kpromoted_mig_cold_not_accessed1373kpromoted_mig_candidate319635kpromoted_mig_promoted319635kpromoted_mig_dropped1599Need to check why is this the case.
Quoted 20 lines
Quoted 17 lines
+
+ /* If the page was hot a while ago, don't promote */ + if ((now -phi->last_update) > 2 *msecs_to_jiffies(KPROMOTED_FREQ_WINDOW)) {
+count_vm_event(KPROMOTED_MIG_COLD_OLD);
+ return false;
+ }
+
+ /* If the page hasn't been accessed enough number of times, don't promote */
+ if (phi->frequency<KPRMOTED_FREQ_THRESHOLD) {
+count_vm_event(KPROMOTED_MIG_COLD_NOT_ACCESSED); + return false;
+ }
+ return true;
+}...
Quoted 12 lines
mm/kpromoted.c line 242
Quoted 10 lines
mm/kpromoted.c new file +10
-0+static int kpromoted (void *p) +{ + pg_data_t *pgdat = (pg_data_t *)p; + struct task_struct *tsk = current; + long timeout = msecs_to_jiffies(KPROMOTE_DELAY); + + const struct cpumask *cpumask = cpumask_of_node(pgdat->node_id); + + if (!cpumask_empty(cpumask)) + set_cpus_allowed_ptr(tsk, cpumask);Explicit cpumasks are not needed if you use
kthread_create_on_node().Thanks, will incorporate.
Regards,
Bharata.
Re: [RFC PATCH 2/4] mm: kpromoted: Hot page info collection and promotion daemon
Quoted 31 lines · Gregory Price
Quoted 30 lines · Bharata B Rao
Quoted 14 lines · Davidlohr Bueso
Quoted 12 lines · Bharata B Rao
mm/kpromoted.c new file +12
-0+static int page_should_be_promoted (struct page_hotness_info *phi) +{ + struct page *page = pfn_to_online_page(phi->pfn); + unsigned long now = jiffies; + struct folio *folio; + + if (!page || is_zone_device_page(page)) + return false; + + folio = page_folio(page); + if (!folio_test_lru(folio)) { + count_vm_event(KPROMOTED_MIG_NON_LRU);Quoted 8 lines
Quoted 6 lines
+ return false;
+ }
+ if (folio_nid(folio)==phi->hot_node) { +count_vm_event(KPROMOTED_MIG_RIGHT_NODE); + return false;
+ }How about using the LRU age itself:
Sounds like a good check for page hotness.
Quoted 3 lines
if (
folio_test_active())
return true;
Quoted 7 lines
Quoted 5 lines
But the numbers I obtained with this check added, didn't really hit this condition all that much. I was running a multi-threaded application that allocates enough memory such that the allocation spills over from DRAM node to the CXL node. Threads keep touching the memory pages in random order.
Is demotion enabled by any chance?
Yes, I thought enabling demotion is required to create enough room in the toptier to handle promotion.
Quoted 3 lines
i.e. are you sure it's actually allocating from CXL and not demoting cold stuff to CXL?
But then I realized that spill over was caused by demotion rather than initial allocation even when I used MPOL_BIND | MPOL_F_NUMA_BALANCING policy with both toptier and CXL node in the nodemask.
Quoted 12 lines
Quoted 12 lines
kpromoted_recorded_accesses960620 /* Number of recorded accesses */kpromoted_recorded_hwhints960620 /* Nr accesses via HW hints, IBS in this case */kpromoted_recorded_pgtscans0kpromoted_record_toptier638006 /* Nr toptier accesses */kpromoted_record_added321234 /* Nr (CXL) accesses that are tracked */kpromoted_record_exists1380kpromoted_mig_right_node0kpromoted_mig_non_lru226kpromoted_mig_lru_active47 /* Number of accesses considered for promotion as determined byfolio_test_active()check */
However disabling demotion has no impact on this number (and hence the folio_test_active() check)
Regards,
Bharata.
Gregory Price
reviewerRe: [RFC PATCH 2/4] mm: kpromoted: Hot page info collection and promotion daemon
Quoted 13 lines · Bharata B Rao
Quoted 12 lines
Quoted 12 lines
kpromoted_recorded_accesses960620 /* Number of recorded accesses */kpromoted_recorded_hwhints960620 /* Nr accesses via HW hints, IBS in this case */kpromoted_recorded_pgtscans0kpromoted_record_toptier638006 /* Nr toptier accesses */kpromoted_record_added321234 /* Nr (CXL) accesses that are tracked */kpromoted_record_exists1380kpromoted_mig_right_node0kpromoted_mig_non_lru226kpromoted_mig_lru_active47 /* Number of accesses considered for promotion as determined byfolio_test_active()check */
Quoted 3 lines
However disabling demotion has no impact on this number (and hence the
folio_test_active()check)
I've been mulling over what's likely to occur when the Low but not Min watermark is hit and reclaim is invoked but without demotion enabled.
I'm wonder if kswapd pushes things like r/o pagecache out, only to have them faulted back into CXL later, while new allocations stick on the main memory.
You might try MPOL_PREFERRED with CXL node as the target instead of bind w/ the local node to at least make sure the system is actually identifying hotness correctly.
~Gregory
Re: [RFC PATCH 2/4] mm: kpromoted: Hot page info collection and promotion daemon
In reply to Bharata B RaoQuoted 24 lines · Bharata B Rao
mm/kpromoted.c new file +24
-0+/* +* Go thro' page hotness information and migrate pages if required. +* +* Promoted pages are not longer tracked in the hot list. +* Cold pages are pruned from the list as well. +* +* TODO: Batching could be done +*/ +static void kpromoted_migrate (pg_data_t *pgdat) +{ +int nid = pgdat->node_id; +struct page_hotness_info *phi; +struct hlist_node *tmp; +int nr_bkts =HASH_SIZE (page_hotness_hash); +int bkt; + +for (bkt =0 ; bkt < nr_bkts; bkt++) { +mutex_lock (&page_hotness_lock[bkt]); +hlist_for_each_entry_safe (phi, tmp, &page_hotness_hash[bkt], hnode) { +if (phi->hot_node != nid) +continue ; + +if (page_should_be_promoted (phi)) { +count_vm_event (KPROMOTED_MIG_CANDIDATE );
mm/kpromoted.c line 105
Quoted 10 lines
mm/kpromoted.c new file +10
-0+if (!kpromote_page (phi)) { +count_vm_event (KPROMOTED_MIG_PROMOTED ); +hlist_del_init (&phi->hnode); +kfree (phi); + } + }else { +/* +* Not a suitable page or cold page, stop tracking it. +* TODO: Identify cold pages and drive demotion? +*/
I don't think kpromoted should drive demotion at all. No one is complaining about migrate in lieu of discard, and there is also proactive reclaim which users can trigger. All the in-kernel problems are wrt promotion. The simpler any of these kthreads are the better.
Quoted 8 lines
mm/kpromoted.c new file +8
-0+count_vm_event (KPROMOTED_MIG_DROPPED ); +hlist_del_init (&phi->hnode); +kfree (phi); + } + } +mutex_unlock (&page_hotness_lock[bkt]); + } +}
Re: [RFC PATCH 2/4] mm: kpromoted: Hot page info collection and promotion daemon
Quoted 33 lines · Davidlohr Bueso
Quoted 31 lines · Bharata B Rao
(quoted diff) +19
-0+/* + * Go thro' page hotness information and migrate pages if required. + * + * Promoted pages are not longer tracked in the hot list. + * Cold pages are pruned from the list as well. + * + * TODO: Batching could be done + */ +static void kpromoted_migrate(pg_data_t *pgdat) +{ + int nid = pgdat->node_id; + struct page_hotness_info *phi; + struct hlist_node *tmp; + int nr_bkts = HASH_SIZE(page_hotness_hash); + int bkt; + + for (bkt = 0; bkt < nr_bkts; bkt++) { + mutex_lock(&page_hotness_lock[bkt]); + hlist_for_each_entry_safe(phi, tmp, &page_hotness_hash[bkt],hnode) {
+ if (phi->hot_node!= nid)
+ continue;
+
+ if (page_should_be_promoted(phi)) {
+count_vm_event(KPROMOTED_MIG_CANDIDATE); + if (!kpromote_page(phi)) {
+count_vm_event(KPROMOTED_MIG_PROMOTED); +hlist_del_init(&phi->hnode); + kfree(phi);
+ }
+ } else {
Quoted 8 lines
Quoted 4 lines
+ /*
+ * Not a suitable page or cold page, stop tracking it. + * TODO: Identify cold pages and drive demotion? + */I don't think kpromoted should drive demotion at all. No one is complaining about migrate in lieu of discard, and there is also proactive reclaim which users can trigger. All the in-kernel problems are wrt promotion. The simpler any of these kthreads are the better.
I was testing on default kernel with NUMA balancing mode 2.
The multi-threaded application allocates memory on DRAM and the allocation spills over to CXL node. The threads keep accessing allocated memory pages in random order.
pgpromote_success 6pgpromote_candidate 745387pgdemote_kswapd 51085pgdemote_direct 10481pgdemote_khugepaged 0numa_pte_updates 27249625numa_huge_pte_updates 0numa_hint_faults 9660745numa_hint_faults_local 0numa_pages_migrated 6numa_node_full 745438pgmigrate_success 2225458pgmigrate_fail 1187349
I hardly see any promotion happening.
In order to check the number of times the toptier node was found to be full when attempting to promote, I added numa_node_full counter like below:
mm/migrate.c +1 -0
@@ -2673,6 +2673,7 @@ int migrate_misplaced_folio_prepare(struct folio *folio,if (!migrate_balanced_pgdat (pgdat, nr_pages)) {int z; +count_vm_event (NUMA_NODE_FULL ); if (!(sysctl_numa_balancing_mode &
NUMA_BALANCING_MEMORY_TIERING))
return -EAGAIN;
for (z = pgdat->nr_zones - 1; z >= 0; z--) {As seen above, numa_node_full 745438. This matches pgpromote_candidate numbers.
I do see counters reporting kswapd-driven and direct demotion as well but does this mean that demotion isn't happening fast enough to cope up with promotion requirement in this high toptier memory pressure situation?
Regards,
Bharata.
Re: [RFC PATCH 2/4] mm: kpromoted: Hot page info collection and promotion daemon
In reply to Bharata B RaoQuoted 5 lines · On Thu, 6 Mar 2025 11:15:30 +0530 Bharata B Rao <[email protected]> wrote:
kpromoted is a kernel daemon that accumulates hot page info from different sources and tries to promote pages from slow tiers to top tiers. One instance of this thread runs on each node that has CPUs.
Firstly, nice work. Much easier to discuss things with an implementation to look at.
I'm looking at this with my hardware hotness unit "hammer" in hand :)
Quoted 5 lines
Subsystems that generate hot page access info can report that to kpromoted via this API:
int kpromoted_record_access(u64 pfn, int nid, int src,
unsigned long time)
This perhaps works as an interface for aggregating methods that produce per access events. Any hardware counter solution is going to give you data that is closer to what you used for the promotion decision.
We might need to aggregate at different levels. So access counting promotes to a hot list and we can inject other events at that level. The data I have from the CXL HMU is typically after an epoch (period of time) these N pages were accessed more than M times. I can sort of map that to the internal storage you have.
Would be good to evaluate approximate trackers on top of access counts. I've no idea if sketches or similar would be efficient enough (they have a bit of a write amplification problem) but they may give good answers with much lower storage cost at the risk of occasionally saying something is hot when it's not.
Quoted 11 lines
@pfn: The PFN of the memory accessed
@nid: The accessing NUMA node ID
@src: The temperature source (subsystem) that generated the
access info
@time: The access time in jiffiesSome temperature sources may not provide the nid from which the page was accessed. This is true for sources that use page table scanning for PTE Accessed bit. Currently the toptier node to which such pages should be promoted to is hard coded.
For those cases (CXL HMU included) maybe we need to consider how to fill in missing node info with at least a vague chance of getting a reasonable target for migration. We can always fall back to random top tier node, or nearest one to where we are coming from (on basis we maybe landed in this node based on a fallback list when the top tier was under memory pressure).
From an interface point of view is that a problem for this layer, or for the underlying tracking mechanism? (maybe with some helpers) Also, see later discussion of consistency of hotness tracking and that the best solution for that differs from that to get potential targets. The answer to Is this page consistently hot?" can be approximated with "Was this page once hot and is it not now cold?"
Access time is something some measurement techniques will only give you wrt to a measurement was in a window (potentially a long one if you are looking for consistent hotness over minutes).
Quoted 8 lines
Also, the access time provided some sources may at best be considered approximate. This is especially true for hot pages detected by PTE A bit scanning.
kpromoted currently maintains the hot PFN records in hash lists hashed by PFN value. Each record stores the following info:
Quoted 9 lines
struct page_hotness_info {
unsigned long pfn;
/* Time when this record was updated last */
unsigned long last_update;
/*
* Number of times this page was accessed in the
* current windowI'd express here how that window was defined (I read on to answer the question I had here at first!)
Quoted 5 lines
*/ int frequency; /* Most recent access time */ unsigned long recency;
Put next to the last_update so all the times are together
Quoted 3 lines
/* Most recent access from this node */ int hot_node;
Probably want to relax the most recent part. I'd guess the ideal here would be if this is the node accessing it the most 'recently'.
Quoted 6 lines
struct hlist_node hnode; };The way in which a page is categorized as hot enough to be promoted is pretty primitive now.
That bit is very hard even if we solve everything else and heavily dependent on workload access pattern stability and migration impact. Maybe for 'very hot' pages a fairly short consistency of hotness period is good enough, but it gets much messier if we care about warm pages. I guess we solve the 'very hot' first though and maybe avoid the phase transition from an application starting to when it is at steady state by considering a wait time for any new userspace process before we consider moving anything?
Also worth noting that the mechanism that makes sense to check if a detected hot page is 'stable hot' might use entirely different tracking approach to that used to find it as a candidate.
Whether that requires passing data between hotness trackers is an interesting question, or whether there is a natural ordering to trackers.
Quoted 16 lines
mm/kpromoted.c new file +10
-0+static int page_should_be_promoted (struct page_hotness_info *phi) +{ +struct page *page =pfn_to_online_page (phi->pfn); +unsigned long now = jiffies; +struct folio *folio; + +if (!page ||is_zone_device_page (page)) +return false; + + folio =page_folio (page);
mm/kpromoted.c line 58
Quoted 10 lines
mm/kpromoted.c new file +10
-0+if (!folio_test_lru (folio)) { +count_vm_event (KPROMOTED_MIG_NON_LRU ); +return false; + } +if (folio_nid (folio) == phi->hot_node) { +count_vm_event (KPROMOTED_MIG_RIGHT_NODE ); +return false; + } + +/* If the page was hot a while ago, don't promote */
/* If the known record of hotness is old, don't promote */ ?
Otherwise this says don't move a page just because it was hot a long time back. Maybe it is still hot and we just don't have an update yet?
Quoted 8 lines
mm/kpromoted.c new file +8
-0+if ((now - phi->last_update) >2 *msecs_to_jiffies (KPROMOTED_FREQ_WINDOW )) { +count_vm_event (KPROMOTED_MIG_COLD_OLD ); +return false; + } + +/* If the page hasn't been accessed enough number of times, don't promote */ +if (phi->frequency <KPRMOTED_FREQ_THRESHOLD ) { +count_vm_event (KPROMOTED_MIG_COLD_NOT_ACCESSED );
mm/kpromoted.c line 76
Quoted 10 lines
mm/kpromoted.c new file +10
-0+return false; + } +return true; +} + +/* +* Go thro' page hotness information and migrate pages if required. +* +* Promoted pages are not longer tracked in the hot list. +* Cold pages are pruned from the list as well.
When we say cold here why did we ever see them?
Quoted 18 lines
+ *
+ * TODO: Batching could be done
+ */mm/kpromoted.c new file +15
-0+static void kpromoted_migrate (pg_data_t *pgdat) +{ +int nid = pgdat->node_id; +struct page_hotness_info *phi; +struct hlist_node *tmp; +int nr_bkts =HASH_SIZE (page_hotness_hash); +int bkt; + +for (bkt =0 ; bkt < nr_bkts; bkt++) { +mutex_lock (&page_hotness_lock[bkt]); +hlist_for_each_entry_safe (phi, tmp, &page_hotness_hash[bkt], hnode) { +if (phi->hot_node != nid) +continue ; + +if (page_should_be_promoted (phi)) {
mm/kpromoted.c line 104
Quoted 10 lines
mm/kpromoted.c new file +10
-0+count_vm_event (KPROMOTED_MIG_CANDIDATE ); +if (!kpromote_page (phi)) { +count_vm_event (KPROMOTED_MIG_PROMOTED ); +hlist_del_init (&phi->hnode); +kfree (phi); + } + }else { +/* +* Not a suitable page or cold page, stop tracking it. +* TODO: Identify cold pages and drive demotion?
Coldness tracking is really different from hotness as we need to track what we didn't see to get the really cold pages. Maybe there is some hint to be had from the exit of this tracker but I'd definitely not try to tackle both ends with one approach!
Quoted 29 lines
mm/kpromoted.c new file +26
-0+*/ +count_vm_event (KPROMOTED_MIG_DROPPED ); +hlist_del_init (&phi->hnode); +kfree (phi); + } + } +mutex_unlock (&page_hotness_lock[bkt]); + } +} +/* +* Called by subsystems that generate page hotness/access information. +* +* Records the memory access info for futher action by kpromoted. +*/ +int kpromoted_record_access (u64 pfn,int nid,int src,unsigned long now) +{ + bkt =hash_min (pfn,KPROMOTED_HASH_ORDER ); +mutex_lock (&page_hotness_lock[bkt]); + phi =kpromoted_lookup (pfn, bkt, now); +if (!phi) { + ret =PTR_ERR (phi); + goto out; + } + +if ((phi->last_update - now) >msecs_to_jiffies (KPROMOTED_FREQ_WINDOW )) { +/* New window */
mm/kpromoted.c line 209
Quoted 10 lines
mm/kpromoted.c new file +10
-0+ phi->frequency =1 ;/* TODO: Factor in the history */ + phi->last_update = now; + }else { + phi->frequency++; + } + phi->recency = now; + +/* +* TODOs: +* 1. Source nid is hard-coded for some temperature sources
Hard coded rather than unknown? I'm curious, what source has that issue?
mm/kpromoted.c line 211
Quoted 2 lines
mm/kpromoted.c new file +2
-0+* 2. Take action if hot_node changes - may be a shared page? +* 3. Maintain node info for every access within the window?
I guess some sort of saturating counter set might not be too bad.
mm/kpromoted.c line 216
Quoted 5 lines
mm/kpromoted.c new file +5
-0+*/ + phi->hot_node = (nid ==NUMA_NO_NODE ) ?1 : nid; +mutex_unlock (&page_hotness_lock[bkt]); +out: +return 0 ;
why store ret and not return it?
Quoted 2 lines
mm/kpromoted.c new file +2
-0+} +
Re: [RFC PATCH 2/4] mm: kpromoted: Hot page info collection and promotion daemon
Quoted 6 lines · Jonathan Cameron
Quoted 3 lines · On Thu, 6 Mar 2025 11:15:30 +0530 Bharata B Rao <[email protected]> wrote:
Subsystems that generate hot page access info can report that to kpromoted via this API:
Quoted 7 lines
Quoted 2 lines
int kpromoted_record_access(u64 pfn, int nid, int src,
unsigned long time)This perhaps works as an interface for aggregating methods that produce per access events. Any hardware counter solution is going to give you data that is closer to what you used for the promotion decision.
Right.
Quoted 7 lines
We might need to aggregate at different levels. So access counting promotes to a hot list and we can inject other events at that level. The data I have from the CXL HMU is typically after an epoch (period of time) these N pages were accessed more than M times. I can sort of map that to the internal storage you have.
Even for IBS source, I am aggregating data in per-cpu buffers before presenting them one by one to kpromoted. Guess CXL HMU aggregated data could be presented in a similar manner.
Quoted 6 lines
Would be good to evaluate approximate trackers on top of access counts. I've no idea if sketches or similar would be efficient enough (they have a bit of a write amplification problem) but they may give good answers with much lower storage cost at the risk of occasionally saying something is hot when it's not.
Could me point me to some information about sketches??
Quoted 13 lines
Quoted 8 lines
@pfn: The PFN of the memory accessed
@nid: The accessing NUMA node ID
@src: The temperature source (subsystem) that generated the
access info
@time: The access time in jiffiesQuoted 4 lines
Some temperature sources may not provide the nid from which the page was accessed. This is true for sources that use page table scanning for PTE Accessed bit. Currently the toptier node to which such pages should be promoted to is hard coded.
Quoted 6 lines
For those cases (CXL HMU included) maybe we need to consider how to fill in missing node info with at least a vague chance of getting a reasonable target for migration. We can always fall back to random top tier node, or nearest one to where we are coming from (on basis we maybe landed in this node based on a fallback list when the top tier was under memory pressure).
Yes. For A-bit scanners, Raghu has devised a scheme to obtain the best possible list of target nodes for promotion. He should be sharing more about it soon.
Quoted 3 lines
From an interface point of view is that a problem for this layer,
or for the underlying tracking mechanism? (maybe with some helpers)
It is not a problem from this interface point of view as this interface expects a nid(or default value) and would use that for promotion. It is up to the underlying tracking mechanism to provide the most appropriate target nid.
Quoted 20 lines
Also, see later discussion of consistency of hotness tracking and that the best solution for that differs from that to get potential targets. The answer to Is this page consistently hot?" can be approximated with "Was this page once hot and is it not now cold?"
Access time is something some measurement techniques will only give you wrt to a measurement was in a window (potentially a long one if you are looking for consistent hotness over minutes).
Quoted 11 lines
Also, the access time provided some sources may at best be considered approximate. This is especially true for hot pages detected by PTE A bit scanning.
kpromoted currently maintains the hot PFN records in hash lists hashed by PFN value. Each record stores the following info:
struct page_hotness_info { unsigned long pfn;
Quoted 8 lines
Quoted 6 lines
/* Time when this record was updated last */ unsigned long last_update; /* * Number of times this page was accessed in the * current windowI'd express here how that window was defined (I read on to answer the question I had here at first!)
Currently the number of accesses that occur within an observation window of 5s are considered for hotness calculation and access count is reset when the window elapses. This needs to factor in history etc.
Quoted 8 lines
Quoted 6 lines
*/ int frequency; /* Most recent access time */ unsigned long recency;Put next to the
last_updateso all the times are together
Sure.
Quoted 8 lines
Quoted 4 lines
/* Most recent access from this node */ int hot_node;Probably want to relax the most recent part. I'd guess the ideal here would be if this is the node accessing it the most 'recently'.
You mean the node that did most number of accesses in the given observation window and not necessarily the last (or most recent) accessed node.
Quoted 17 lines
Quoted 7 lines
struct hlist_node hnode; };The way in which a page is categorized as hot enough to be promoted is pretty primitive now.
That bit is very hard even if we solve everything else and heavily dependent on workload access pattern stability and migration impact. Maybe for 'very hot' pages a fairly short consistency of hotness period is good enough, but it gets much messier if we care about warm pages. I guess we solve the 'very hot' first though and maybe avoid the phase transition from an application starting to when it is at steady state by considering a wait time for any new userspace process before we consider moving anything?
Quoted 6 lines
Also worth noting that the mechanism that makes sense to check if a detected hot page is 'stable hot' might use entirely different tracking approach to that used to find it as a candidate.
Whether that requires passing data between hotness trackers is an interesting question, or whether there is a natural ordering to trackers.
I was envisioning that different hotness trackers would reinforce the page hotness by reporting the same to kpromoted and there would be no need to again pass data between hotness trackers.
Quoted 32 lines
Quoted 19 lines
mm/kpromoted.c new file +10
-0+static int page_should_be_promoted (struct page_hotness_info *phi) +{ +struct page *page =pfn_to_online_page (phi->pfn); +unsigned long now = jiffies; +struct folio *folio; + +if (!page ||is_zone_device_page (page)) +return false; + + folio =page_folio (page);mm/kpromoted.c line 58
Quoted 10 lines
mm/kpromoted.c new file +10
-0+if (!folio_test_lru (folio)) { +count_vm_event (KPROMOTED_MIG_NON_LRU ); +return false; + } +if (folio_nid (folio) == phi->hot_node) { +count_vm_event (KPROMOTED_MIG_RIGHT_NODE ); +return false; + } + +/* If the page was hot a while ago, don't promote */ /* If the known record of hotness is old, don't promote */ ?
Quoted 2 lines
Otherwise this says don't move a page just because it was hot a long time back. Maybe it is still hot and we just don't have an update yet?
Agreed.
Quoted 13 lines
Quoted 13 lines
mm/kpromoted.c new file +12
-0+if ((now - phi->last_update) >2 *msecs_to_jiffies (KPROMOTED_FREQ_WINDOW )) { +count_vm_event (KPROMOTED_MIG_COLD_OLD ); +return false; + } + +/* If the page hasn't been accessed enough number of times, don't promote */ +if (phi->frequency <KPRMOTED_FREQ_THRESHOLD ) { +count_vm_event (KPROMOTED_MIG_COLD_NOT_ACCESSED ); +return false; + } +return true; +}
Quoted 8 lines
mm/kpromoted.c line 76
Quoted 6 lines
mm/kpromoted.c new file +6
-0+ +/* +* Go thro' page hotness information and migrate pages if required. +* +* Promoted pages are not longer tracked in the hot list. +* Cold pages are pruned from the list as well. When we say cold here why did we ever see them?
Those hot pages that couldn't be migrated for different reasons are no longer tracked by kpromoted and I called such pages as "cold". Guess not the right nomenclature to represent them.
Quoted 26 lines
Quoted 26 lines
+ *
+ * TODO: Batching could be done
+ */mm/kpromoted.c new file +22
-0+static void kpromoted_migrate (pg_data_t *pgdat) +{ +int nid = pgdat->node_id; +struct page_hotness_info *phi; +struct hlist_node *tmp; +int nr_bkts =HASH_SIZE (page_hotness_hash); +int bkt; + +for (bkt =0 ; bkt < nr_bkts; bkt++) { +mutex_lock (&page_hotness_lock[bkt]); +hlist_for_each_entry_safe (phi, tmp, &page_hotness_hash[bkt], hnode) { +if (phi->hot_node != nid) +continue ; + +if (page_should_be_promoted (phi)) { +count_vm_event (KPROMOTED_MIG_CANDIDATE ); +if (!kpromote_page (phi)) { +count_vm_event (KPROMOTED_MIG_PROMOTED ); +hlist_del_init (&phi->hnode); +kfree (phi); + } + }else {
Quoted 8 lines
mm/kpromoted.c line 104
Quoted 3 lines
mm/kpromoted.c new file +3
-0+/* +* Not a suitable page or cold page, stop tracking it. +* TODO: Identify cold pages and drive demotion? Coldness tracking is really different from hotness as we need to track what we didn't see to get the really cold pages. Maybe there is some hint to be had from the exit of this tracker but I'd definitely not try to tackle both ends with one approach!
Okay.
Quoted 34 lines
Quoted 34 lines
mm/kpromoted.c new file +30
-0+*/ +count_vm_event (KPROMOTED_MIG_DROPPED ); +hlist_del_init (&phi->hnode); +kfree (phi); + } + } +mutex_unlock (&page_hotness_lock[bkt]); + } +} +/* +* Called by subsystems that generate page hotness/access information. +* +* Records the memory access info for futher action by kpromoted. +*/ +int kpromoted_record_access (u64 pfn,int nid,int src,unsigned long now) +{ + bkt =hash_min (pfn,KPROMOTED_HASH_ORDER ); +mutex_lock (&page_hotness_lock[bkt]); + phi =kpromoted_lookup (pfn, bkt, now); +if (!phi) { + ret =PTR_ERR (phi); + goto out; + } + +if ((phi->last_update - now) >msecs_to_jiffies (KPROMOTED_FREQ_WINDOW )) { +/* New window */ + phi->frequency =1 ;/* TODO: Factor in the history */ + phi->last_update = now; + }else { + phi->frequency++;
Quoted 8 lines
mm/kpromoted.c line 209
Quoted 6 lines
mm/kpromoted.c new file +6
-0+ } + phi->recency = now; + +/* +* TODOs: +* 1. Source nid is hard-coded for some temperature sources Hard coded rather than unknown? I'm curious, what source has that issue?
I meant that source didn't provide a nid and hence kpromoted ended up promoting to a fixed (hard-coded for now) toptier node.
Quoted 5 lines
mm/kpromoted.c line 211
Quoted 3 lines
mm/kpromoted.c new file +2
-0+* 2. Take action if hot_node changes - may be a shared page? +* 3. Maintain node info for every access within the window? I guess some sort of saturating counter set might not be too bad.
Yes.
Quoted 8 lines
mm/kpromoted.c line 216
Quoted 6 lines
mm/kpromoted.c new file +5
-0+*/ + phi->hot_node = (nid ==NUMA_NO_NODE ) ?1 : nid; +mutex_unlock (&page_hotness_lock[bkt]); +out: +return 0 ;why store ret and not return it?
Will fix.
Thanks for your review!
Regards,
Bharata.
Re: [RFC PATCH 2/4] mm: kpromoted: Hot page info collection and promotion daemon
Quoted 25 lines · On Tue, 18 Mar 2025 09:39:17 +0530 Bharata B Rao <[email protected]> wrote:
Quoted 6 lines · Jonathan Cameron
Quoted 3 lines · On Thu, 6 Mar 2025 11:15:30 +0530 Bharata B Rao <[email protected]> wrote:
Subsystems that generate hot page access info can report that to kpromoted via this API:
Quoted 7 lines
Quoted 2 lines
int kpromoted_record_access(u64 pfn, int nid, int src,
unsigned long time)This perhaps works as an interface for aggregating methods that produce per access events. Any hardware counter solution is going to give you data that is closer to what you used for the promotion decision.
Right.
Quoted 7 lines
We might need to aggregate at different levels. So access counting promotes to a hot list and we can inject other events at that level. The data I have from the CXL HMU is typically after an epoch (period of time) these N pages were accessed more than M times. I can sort of map that to the internal storage you have.
Quoted 3 lines
Even for IBS source, I am aggregating data in per-cpu buffers before presenting them one by one to kpromoted. Guess CXL HMU aggregated data could be presented in a similar manner.
The nature of the data maybe a bit different but certainly should be able to find somewhere in the stack!
Quoted 9 lines
Quoted 7 lines
Would be good to evaluate approximate trackers on top of access counts. I've no idea if sketches or similar would be efficient enough (they have a bit of a write amplification problem) but they may give good answers with much lower storage cost at the risk of occasionally saying something is hot when it's not.
Could me point me to some information about sketches??
https://en.wikipedia.org/wiki/Count%E2%80%93min_sketch
Is a good starting point but there are lots of related techniques that trade off good statistical properties against complexity etc.
Roughly speaking you combine a sorted list of the very hottest with a small number of different hash tables (the sketch) that let you get an estimate of how hot things are that have dropped off your hottest list (or not yet gotten hot enough to get into it).
Looking for literature for top-k algorithms will find you more references though not all are light weight enough to be of interest here.
Quoted 21 lines
Quoted 14 lines
Quoted 9 lines
@pfn: The PFN of the memory accessed
@nid: The accessing NUMA node ID
@src: The temperature source (subsystem) that generated the
access info
@time: The access time in jiffiesQuoted 4 lines
Some temperature sources may not provide the nid from which the page was accessed. This is true for sources that use page table scanning for PTE Accessed bit. Currently the toptier node to which such pages should be promoted to is hard coded.
Quoted 6 lines
For those cases (CXL HMU included) maybe we need to consider how to fill in missing node info with at least a vague chance of getting a reasonable target for migration. We can always fall back to random top tier node, or nearest one to where we are coming from (on basis we maybe landed in this node based on a fallback list when the top tier was under memory pressure).
Quoted 3 lines
Yes. For A-bit scanners, Raghu has devised a scheme to obtain the best possible list of target nodes for promotion. He should be sharing more about it soon.
Excellent - look forward to seeing that. Can think of a few possibilities on how to get that data efficiently so I'm curious what Raghu has chosen.
Quoted 9 lines
Quoted 4 lines
From an interface point of view is that a problem for this layer,
or for the underlying tracking mechanism? (maybe with some helpers)It is not a problem from this interface point of view as this interface expects a nid(or default value) and would use that for promotion. It is up to the underlying tracking mechanism to provide the most appropriate target nid.
I was wondering if there is some sharing to do, so whether we push Raghu's means of getting a target node down into the tracker implementation or use it to fill in messing info at this layer. Will depend a bit on how that technique works perhaps.
Quoted 27 lines
Quoted 27 lines
Also, see later discussion of consistency of hotness tracking and that the best solution for that differs from that to get potential targets. The answer to Is this page consistently hot?" can be approximated with "Was this page once hot and is it not now cold?"
Access time is something some measurement techniques will only give you wrt to a measurement was in a window (potentially a long one if you are looking for consistent hotness over minutes).
Quoted 17 lines
Also, the access time provided some sources may at best be considered approximate. This is especially true for hot pages detected by PTE A bit scanning.
kpromoted currently maintains the hot PFN records in hash lists hashed by PFN value. Each record stores the following info:
struct page_hotness_info { unsigned long pfn; /* Time when this record was updated last */ unsigned long last_update; /* * Number of times this page was accessed in the * current window
Quoted 6 lines
Quoted 2 lines
I'd express here how that window was defined (I read on to answer the question I had here at first!)
Currently the number of accesses that occur within an observation window of 5s are considered for hotness calculation and access count is reset when the window elapses. This needs to factor in history etc.
Just add that to the comment here perhaps.
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
/* Most recent access from this node */ int hot_node;
Quoted 7 lines
Quoted 3 lines
Probably want to relax the most recent part. I'd guess the ideal here would be if this is the node accessing it the most 'recently'.
You mean the node that did most number of accesses in the given observation window and not necessarily the last (or most recent) accessed node.
yes. Though maybe weighted in some fashion for recency? Something cheap to do that approximates that such as small saturating counters with aging..
Quoted 22 lines
Quoted 22 lines
Quoted 8 lines
struct hlist_node hnode; };The way in which a page is categorized as hot enough to be promoted is pretty primitive now.
That bit is very hard even if we solve everything else and heavily dependent on workload access pattern stability and migration impact. Maybe for 'very hot' pages a fairly short consistency of hotness period is good enough, but it gets much messier if we care about warm pages. I guess we solve the 'very hot' first though and maybe avoid the phase transition from an application starting to when it is at steady state by considering a wait time for any new userspace process before we consider moving anything?
Also worth noting that the mechanism that makes sense to check if a detected hot page is 'stable hot' might use entirely different tracking approach to that used to find it as a candidate.
Quoted 6 lines
Quoted 2 lines
Whether that requires passing data between hotness trackers is an interesting question, or whether there is a natural ordering to trackers.
I was envisioning that different hotness trackers would reinforce the page hotness by reporting the same to kpromoted and there would be no need to again pass data between hotness trackers.
What makes me wonder about that is the question of stability of hotness. It is a really bad idea to move data based on a short sample - cost is huge and quite a bit of data is only briefly hot - moving it to fast memory too early just results in bouncing. There are probably heuristics we can apply on process age etc that will help, but generally we can't assume programs don't have multiple phases with very different access characteristics.
The different tracking approaches have different sweet spots for short vs long tracking. So it might be a case of one method, e.g. a hotness tracker is only suitable for monitoring a short time period
(in the simplest sense, counters saturate if you run too long). Don't read that to generally though as it's not a universal characteristic and depends on the implementation used, but it is definitely true of some potential implementations.
Having gotten a list of 1000+ candidate pages that might be worth moving, we could use access bits to check it's still accessed reasonsably frequently over the next minute. That can be much lower cost than an access tracker that is looking for 'hottest'.
Where all these trade offs with timing occur is tricky and workload dependent. So figuring out how to autotune will be a challenging.
Quoted 34 lines
Quoted 25 lines
Quoted 25 lines
mm/kpromoted.c new file +14
-0+ bkt =hash_min (pfn,KPROMOTED_HASH_ORDER ); +mutex_lock (&page_hotness_lock[bkt]); + phi =kpromoted_lookup (pfn, bkt, now); +if (!phi) { + ret =PTR_ERR (phi); + goto out; + } + +if ((phi->last_update - now) >msecs_to_jiffies (KPROMOTED_FREQ_WINDOW )) { +/* New window */ + phi->frequency =1 ;/* TODO: Factor in the history */ + phi->last_update = now; + }else { + phi->frequency++;Quoted 8 lines
mm/kpromoted.c line 209
Quoted 6 lines
mm/kpromoted.c new file +6
-0+ } + phi->recency = now; + +/* +* TODOs: +* 1. Source nid is hard-coded for some temperature sources Hard coded rather than unknown? I'm curious, what source has that issue?
Quoted 2 lines
I meant that source didn't provide a nid and hence kpromoted ended up promoting to a fixed (hard-coded for now) toptier node.
Sure. Unknown nid makes sense here.
Thanks,
Jonathan
Re: [RFC PATCH 2/4] mm: kpromoted: Hot page info collection and promotion daemon
In reply to Bharata B RaoQuoted 5 lines · Bharata B Rao
kpromoted is a kernel daemon that accumulates hot page info from different sources and tries to promote pages from slow tiers to top tiers. One instance of this thread runs on each node that has CPUs.
Could you please elaborate on what is slow vs top tier? A top tier uses adist (which is a combination of bandwidth and latency), so I am not sure the terminology here holds.
Quoted 6 lines
Subsystems that generate hot page access info can report that to kpromoted via this API:
int kpromoted_record_access(u64 pfn, int nid, int src,
unsigned long time)
Quoted 7 lines
@pfn: The PFN of the memory accessed
@nid: The accessing NUMA node ID
@src: The temperature source (subsystem) that generated the
access info
@time: The access time in jiffiesSome temperature sources may not provide the nid from which
What is a temperature source?
Quoted 4 lines
the page was accessed. This is true for sources that use page table scanning for PTE Accessed bit. Currently the toptier node to which such pages should be promoted to is hard coded.
What would it take to make this flexible?
Quoted 62 lines
Also, the access time provided some sources may at best be considered approximate. This is especially true for hot pages detected by PTE A bit scanning.
kpromoted currently maintains the hot PFN records in hash lists hashed by PFN value. Each record stores the following info:
struct page_hotness_info { unsigned long pfn; /* Time when this record was updated last */ unsigned long last_update; /* * Number of times this page was accessed in the * current window */ int frequency; /* Most recent access time */ unsigned long recency; /* Most recent access from this node */ int hot_node; struct hlist_node hnode; };The way in which a page is categorized as hot enough to be promoted is pretty primitive now.
Signed-off-by Bharata B Rao <[email protected]>--- include/linux/kpromoted.h | 54 ++++++ include/linux/mmzone.h | 4 + include/linux/vm_event_item.h | 13 ++ mm/Kconfig | 7 + mm/Makefile | 1 + mm/kpromoted.c | 305 ++++++++++++++++++++++++++++++++++ mm/mm_init.c | 10 ++ mm/vmstat.c | 13 ++ 8 files changed, 407 insertions(+) create mode 100644 include/linux/kpromoted.h create mode 100644 mm/kpromoted.cinclude/linux/kpromoted.h new file +11
-0@@ -0,0 +1,54 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_KPROMOTED_H +#define _LINUX_KPROMOTED_H + +#include <linux/types.h> +#include <linux/init.h> +#include <linux/workqueue_types.h> + +/* Page hotness temperature sources */ +enum kpromoted_src { + KPROMOTED_HW_HINTS,
include/linux/kpromoted.h line 21
Quoted 10 lines
include/linux/kpromoted.h new file +10
-0+ KPROMOTED_PGTABLE_SCAN, +}; + +#ifdef CONFIG_KPROMOTED + +#define KPROMOTED_FREQ_WINDOW (5 * MSEC_PER_SEC) + +/* 2 accesses within a window will make the page a promotion candidate */ +#define KPRMOTED_FREQ_THRESHOLD 2 +
Were these value derived empirically?
Quoted 118 lines
include/linux/kpromoted.h new file +33
-0+#define KPROMOTED_HASH_ORDER 16 + +struct page_hotness_info { +unsigned long pfn; + +/* Time when this record was updated last */ +unsigned long last_update; + +/* +* Number of times this page was accessed in the +* current window +*/ +int frequency; + +/* Most recent access time */ +unsigned long recency; + +/* Most recent access from this node */ +int hot_node; +struct hlist_node hnode; +}; + +#define KPROMOTE_DELAY MSEC_PER_SEC + +int kpromoted_record_access (u64 pfn,int nid,int src,unsigned long now); +#else +static inline int kpromoted_record_access (u64 pfn,int nid,int src, +unsigned long now) +{ +return 0 ; +} +#endif /* CONFIG_KPROMOTED */ +#endif /* _LINUX_KPROMOTED_H */ include/linux/mmzone.h +4
-0@@ -1459,6 +1459,10 @@ typedef struct pglist_data {#ifdef CONFIG_MEMORY_FAILURE struct memory_failure_stats mf_stats;#endif +#ifdef CONFIG_KPROMOTED +struct task_struct *kpromoted; +wait_queue_head_t kpromoted_wait; +#endif } pg_data_t;#define node_present_pages (nid) (NODE_DATA(nid)->node_present_pages)include/linux/vm_event_item.h +13
-0@@ -182,6 +182,19 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, KSTACK_REST,#endif #endif /* CONFIG_DEBUG_STACK_USAGE */ +KPROMOTED_RECORDED_ACCESSES , +KPROMOTED_RECORD_HWHINTS , +KPROMOTED_RECORD_PGTSCANS , +KPROMOTED_RECORD_TOPTIER , +KPROMOTED_RECORD_ADDED , +KPROMOTED_RECORD_EXISTS , +KPROMOTED_MIG_RIGHT_NODE , +KPROMOTED_MIG_NON_LRU , +KPROMOTED_MIG_COLD_OLD , +KPROMOTED_MIG_COLD_NOT_ACCESSED , +KPROMOTED_MIG_CANDIDATE , +KPROMOTED_MIG_PROMOTED , +KPROMOTED_MIG_DROPPED ,NR_VM_EVENT_ITEMS };mm/Kconfig +7
-0@@ -1358,6 +1358,13 @@ config PT_RECLAIM Note: now only empty user PTE page table pages will be reclaimed. +config KPROMOTED + bool "Kernel hot page promotion daemon" + def_bool y + depends on NUMA && MIGRATION && MMU + help + Promote hot pages from lower tier to top tier by using the + memory access information provided by various sources. source "mm/damon/Kconfig"mm/Makefile +1
-0@@ -147,3 +147,4 @@ obj-$(CONFIG_SHRINKER_DEBUG) += shrinker_debug.o obj-$(CONFIG_EXECMEM) += execmem.o obj-$(CONFIG_TMPFS_QUOTA) += shmem_quota.o obj-$(CONFIG_PT_RECLAIM) += pt_reclaim.o +obj-$(CONFIG_KPROMOTED) += kpromoted.omm/kpromoted.c new file +13
-0@@ -0,0 +1,305 @@ +// SPDX-License-Identifier: GPL-2.0 +/* +* kpromoted is a kernel thread that runs on each node that has CPU i,e., +* on regular nodes. +* +* Maintains list of hot pages from lower tiers and promotes them. +*/ +#include <linux/kpromoted.h> +#include <linux/kthread.h> +#include <linux/mutex.h> +#include <linux/mmzone.h> +#include <linux/migrate.h> +#include <linux/memory-tiers.h>
mm/kpromoted.c line 23
Quoted 10 lines
mm/kpromoted.c new file +10
-0+#include <linux/slab.h> +#include <linux/sched.h> +#include <linux/cpuhotplug.h> +#include <linux/hashtable.h> + +static DEFINE_HASHTABLE(page_hotness_hash, KPROMOTED_HASH_ORDER); +static struct mutex page_hotness_lock[1UL <<KPROMOTED_HASH_ORDER ]; + +static int kpromote_page (struct page_hotness_info *phi) +{
Why not just call it kpromote_folio?
mm/kpromoted.c line 29
Quoted 6 lines
mm/kpromoted.c new file +6
-0+struct page *page =pfn_to_page (phi->pfn); +struct folio *folio; +int ret; + +if (!page) +return 1 ;
Do we need to check for is_zone_device_page() here?
mm/kpromoted.c line 37
Quoted 8 lines
mm/kpromoted.c new file +8
-0+ + folio =page_folio (page); + ret =migrate_misplaced_folio_prepare (folio,NULL , phi->hot_node); +if (ret) +return 1 ; + +return migrate_misplaced_folio (folio, phi->hot_node); +}
Could you please document the assumptions for kpromote_page(), what locks should be held? Does the ref count need to be incremented?
Quoted 13 lines
mm/kpromoted.c new file +13
-0+ +static int page_should_be_promoted (struct page_hotness_info *phi) +{ +struct page *page =pfn_to_online_page (phi->pfn); +unsigned long now = jiffies; +struct folio *folio; + +if (!page ||is_zone_device_page (page)) +return false; + + folio =page_folio (page); +if (!folio_test_lru (folio)) { +count_vm_event (KPROMOTED_MIG_NON_LRU );
mm/kpromoted.c line 60
Quoted 10 lines
mm/kpromoted.c new file +10
-0+return false; + } +if (folio_nid (folio) == phi->hot_node) { +count_vm_event (KPROMOTED_MIG_RIGHT_NODE ); +return false; + } + +/* If the page was hot a while ago, don't promote */ +if ((now - phi->last_update) >2 *msecs_to_jiffies (KPROMOTED_FREQ_WINDOW )) { +count_vm_event (KPROMOTED_MIG_COLD_OLD );
Shouldn't we update phi->last_update here?
Quoted 38 lines
mm/kpromoted.c new file +38
-0+return false; + } + +/* If the page hasn't been accessed enough number of times, don't promote */ +if (phi->frequency <KPRMOTED_FREQ_THRESHOLD ) { +count_vm_event (KPROMOTED_MIG_COLD_NOT_ACCESSED ); +return false; + } +return true; +} + +/* +* Go thro' page hotness information and migrate pages if required. +* +* Promoted pages are not longer tracked in the hot list. +* Cold pages are pruned from the list as well. +* +* TODO: Batching could be done +*/ +static void kpromoted_migrate (pg_data_t *pgdat) +{ +int nid = pgdat->node_id; +struct page_hotness_info *phi; +struct hlist_node *tmp; +int nr_bkts =HASH_SIZE (page_hotness_hash); +int bkt; + +for (bkt =0 ; bkt < nr_bkts; bkt++) { +mutex_lock (&page_hotness_lock[bkt]); +hlist_for_each_entry_safe (phi, tmp, &page_hotness_hash[bkt], hnode) { +if (phi->hot_node != nid) +continue ; + +if (page_should_be_promoted (phi)) { +count_vm_event (KPROMOTED_MIG_CANDIDATE ); +if (!kpromote_page (phi)) { +count_vm_event (KPROMOTED_MIG_PROMOTED ); +hlist_del_init (&phi->hnode);
mm/kpromoted.c line 108
Quoted 10 lines
mm/kpromoted.c new file +10
-0+kfree (phi); + } + }else { +/* +* Not a suitable page or cold page, stop tracking it. +* TODO: Identify cold pages and drive demotion? +*/ +count_vm_event (KPROMOTED_MIG_DROPPED ); +hlist_del_init (&phi->hnode); +kfree (phi);
Won't existing demotion already handle this?
mm/kpromoted.c line 114
Quoted 6 lines
mm/kpromoted.c new file +6
-0+ } + } +mutex_unlock (&page_hotness_lock[bkt]); + } +} +
It sounds like NUMA balancing, promotion and demotion can all act on parallel on these folios, if not could you clarify their relationship and dependency?
Quoted 89 lines
mm/kpromoted.c new file +89
-0+static struct page_hotness_info *__kpromoted_lookup (unsigned long pfn,int bkt) +{ +struct page_hotness_info *phi; + +hlist_for_each_entry (phi, &page_hotness_hash[bkt], hnode) { +if (phi->pfn == pfn) +return phi; + } +return NULL ; +} + +static struct page_hotness_info *kpromoted_lookup (unsigned long pfn,int bkt,unsigned long now) +{ +struct page_hotness_info *phi; + + phi =__kpromoted_lookup (pfn, bkt); +if (!phi) { + phi =kzalloc (sizeof (struct page_hotness_info ),GFP_KERNEL ); +if (!phi) +return ERR_PTR (-ENOMEM ); + + phi->pfn = pfn; + phi->frequency =1 ; + phi->last_update = now; + phi->recency = now; +hlist_add_head (&phi->hnode, &page_hotness_hash[bkt]); +count_vm_event (KPROMOTED_RECORD_ADDED ); + }else { +count_vm_event (KPROMOTED_RECORD_EXISTS ); + } +return phi; +} + +/* +* Called by subsystems that generate page hotness/access information. +* +* Records the memory access info for futher action by kpromoted. +*/ +int kpromoted_record_access (u64 pfn,int nid,int src,unsigned long now) +{ +struct page_hotness_info *phi; +struct page *page; +struct folio *folio; +int ret, bkt; + +count_vm_event (KPROMOTED_RECORDED_ACCESSES ); + +switch (src) { +case KPROMOTED_HW_HINTS : +count_vm_event (KPROMOTED_RECORD_HWHINTS ); +break ; +case KPROMOTED_PGTABLE_SCAN : +count_vm_event (KPROMOTED_RECORD_PGTSCANS ); +break ; +default : +break ; + } + +/* +* Record only accesses from lower tiers. +* Assuming node having CPUs as toptier for now. +*/ +if (node_is_toptier (pfn_to_nid (pfn))) { +count_vm_event (KPROMOTED_RECORD_TOPTIER ); +return 0 ; + } + + page =pfn_to_online_page (pfn); +if (!page ||is_zone_device_page (page)) +return 0 ; + + folio =page_folio (page); +if (!folio_test_lru (folio)) +return 0 ; + + bkt =hash_min (pfn,KPROMOTED_HASH_ORDER ); +mutex_lock (&page_hotness_lock[bkt]); + phi =kpromoted_lookup (pfn, bkt, now); +if (!phi) { + ret =PTR_ERR (phi); + goto out; + } + +if ((phi->last_update - now) >msecs_to_jiffies (KPROMOTED_FREQ_WINDOW )) { +/* New window */ + phi->frequency =1 ;/* TODO: Factor in the history */ + phi->last_update = now; + }else { + phi->frequency++;
mm/kpromoted.c line 213
Quoted 10 lines
mm/kpromoted.c new file +10
-0+ } + phi->recency = now; + +/* +* TODOs: +* 1. Source nid is hard-coded for some temperature sources +* 2. Take action if hot_node changes - may be a shared page? +* 3. Maintain node info for every access within the window? +*/ + phi->hot_node = (nid ==NUMA_NO_NODE ) ?1 : nid;
I don't understand why nid needs to be 1 if nid is NUMA_NODE_ID? Does it mean that it's being promoted to the top tier, the mix of hot_node, tier and nid is not very clear here.
Quoted 78 lines
mm/kpromoted.c new file +78
-0+mutex_unlock (&page_hotness_lock[bkt]); +out: +return 0 ; +} + +/* +* Go through the accumulated mem_access_info and migrate +* pages if required. +*/ +static void kpromoted_do_work (pg_data_t *pgdat) +{ +kpromoted_migrate (pgdat); +} + +static inline bool kpromoted_work_requested (pg_data_t *pgdat) +{ +return false; +} + +static int kpromoted (void *p) +{ +pg_data_t *pgdat = (pg_data_t *)p; +struct task_struct *tsk = current; +long timeout =msecs_to_jiffies (KPROMOTE_DELAY ); + +const struct cpumask *cpumask =cpumask_of_node (pgdat->node_id); + +if (!cpumask_empty (cpumask)) +set_cpus_allowed_ptr (tsk, cpumask); + +while (!kthread_should_stop ()) { +wait_event_timeout (pgdat->kpromoted_wait, +kpromoted_work_requested (pgdat), timeout); +kpromoted_do_work (pgdat); + } +return 0 ; +} + +static void kpromoted_run (int nid) +{ +pg_data_t *pgdat =NODE_DATA (nid); + +if (pgdat->kpromoted) +return ; + + pgdat->kpromoted =kthread_run (kpromoted, pgdat,"kpromoted%d" , nid); +if (IS_ERR (pgdat->kpromoted)) { +pr_err ("Failed to start kpromoted on node %d\n" , nid); + pgdat->kpromoted =NULL ; + } +} + +static int kpromoted_cpu_online (unsigned int cpu) +{ +int nid; + +for_each_node_state (nid,N_CPU ) { +pg_data_t *pgdat =NODE_DATA (nid); +const struct cpumask *mask; + + mask =cpumask_of_node (pgdat->node_id); + +if (cpumask_any_and (cpu_online_mask, mask) < nr_cpu_ids) +/* One of our CPUs online: restore mask */ +if (pgdat->kpromoted) +set_cpus_allowed_ptr (pgdat->kpromoted, mask); + } +return 0 ; +} + +static int __init kpromoted_init (void ) +{ +int nid, ret, i; + + ret =cpuhp_setup_state_nocalls (CPUHP_AP_ONLINE_DYN , +"mm/promotion:online" , + kpromoted_cpu_online,NULL ); +if (ret <0 ) {
mm/kpromoted.c line 301
Quoted 10 lines
mm/kpromoted.c new file +10
-0+pr_err ("kpromoted: failed to register hotplug callbacks.\n" ); +return ret; + } + +for (i =0 ; i < (1UL <<KPROMOTED_HASH_ORDER ); i++) +mutex_init (&page_hotness_lock[i]); + +for_each_node_state (nid,N_CPU ) +kpromoted_run (nid); +
I think we need a dynamic way to disabling promotion at run time as well, right?
Quoted 56 lines
mm/kpromoted.c new file +4
-0+return 0 ; +} + +subsys_initcall (kpromoted_init)mm/mm_init.c +10
-0@@ -1362,6 +1362,15 @@ static void pgdat_init_kcompactd(struct pglist_data *pgdat)static void pgdat_init_kcompactd (struct pglist_data *pgdat) {}#endif +#ifdef CONFIG_KPROMOTED +static void pgdat_init_kpromoted (struct pglist_data *pgdat) +{ +init_waitqueue_head (&pgdat->kpromoted_wait); +} +#else +static void pgdat_init_kpromoted (struct pglist_data *pgdat) {} +#endif +static void __meminit pgdat_init_internals (struct pglist_data *pgdat) {int i;@@ -1371,6 +1380,7 @@ static void __meminit pgdat_init_internals(struct pglist_data *pgdat)pgdat_init_split_queue (pgdat);pgdat_init_kcompactd (pgdat); +pgdat_init_kpromoted (pgdat);init_waitqueue_head (&pgdat->kswapd_wait);init_waitqueue_head (&pgdat->pfmemalloc_wait);mm/vmstat.c +13
-0@@ -1466,6 +1466,19 @@ const char * const vmstat_text[] = { "kstack_rest",#endif #endif +"kpromoted_recorded_accesses" , +"kpromoted_recorded_hwhints" , +"kpromoted_recorded_pgtscans" , +"kpromoted_record_toptier" , +"kpromoted_record_added" , +"kpromoted_record_exists" , +"kpromoted_mig_right_node" , +"kpromoted_mig_non_lru" , +"kpromoted_mig_cold_old" , +"kpromoted_mig_cold_not_accessed" , +"kpromoted_mig_candidate" , +"kpromoted_mig_promoted" , + "kpromoted_mig_dropped", #endif /* CONFIG_VM_EVENT_COUNTERS || CONFIG_MEMCG */ };#endif /* CONFIG_PROC_FS || CONFIG_SYSFS || CONFIG_NUMA || CONFIG_MEMCG */
Re: [RFC PATCH 2/4] mm: kpromoted: Hot page info collection and promotion daemon
Hi Balbir,
Sorry for the delay in response and thanks for the review...
Quoted 10 lines · Balbir Singh
Quoted 5 lines · Bharata B Rao
kpromoted is a kernel daemon that accumulates hot page info from different sources and tries to promote pages from slow tiers to top tiers. One instance of this thread runs on each node that has CPUs.
Could you please elaborate on what is slow vs top tier? A top tier uses adist (which is a combination of bandwidth and latency), so I am not sure the terminology here holds.
Slow is used to mean bottom tiers here as determined by the memory tiering hierarchy.
Quoted 13 lines
Quoted 13 lines
Subsystems that generate hot page access info can report that to kpromoted via this API:
int kpromoted_record_access(u64 pfn, int nid, int src,
unsigned long time)@pfn: The PFN of the memory accessed
@nid: The accessing NUMA node ID
@src: The temperature source (subsystem) that generated the
access info
@time: The access time in jiffies
Quoted 3 lines
Quoted 1 line
Some temperature sources may not provide the nid from which
What is a temperature source?
Temperature source is a term used to refer to the subsystem that generates memory access information. For e.g. LRU subsystem that scans page tables for Accessed bit becomes a source.
Quoted 7 lines
Quoted 5 lines
the page was accessed. This is true for sources that use page table scanning for PTE Accessed bit. Currently the toptier node to which such pages should be promoted to is hard coded.
What would it take to make this flexible?
The context here is that sources that provide access information by scanning the PTE A bit wouldn't know from which node the access was done. Same is the case for kmmscand approach though Raghu has some heuristics to deduce the best possible toptier node to which a given page should be promoted. More details at https://lore.kernel.org/linux-mm/[email protected]/
What kpromoted did for such cases is to just promote the pages to a node whose nid has been hard-coded for now (like 0 or 1 etc)
Quoted 67 lines
Quoted 67 lines
Also, the access time provided some sources may at best be considered approximate. This is especially true for hot pages detected by PTE A bit scanning.
kpromoted currently maintains the hot PFN records in hash lists hashed by PFN value. Each record stores the following info:
struct page_hotness_info { unsigned long pfn; /* Time when this record was updated last */ unsigned long last_update; /* * Number of times this page was accessed in the * current window */ int frequency; /* Most recent access time */ unsigned long recency; /* Most recent access from this node */ int hot_node; struct hlist_node hnode; };The way in which a page is categorized as hot enough to be promoted is pretty primitive now.
Signed-off-by Bharata B Rao <[email protected]>--- include/linux/kpromoted.h | 54 ++++++ include/linux/mmzone.h | 4 + include/linux/vm_event_item.h | 13 ++ mm/Kconfig | 7 + mm/Makefile | 1 + mm/kpromoted.c | 305 ++++++++++++++++++++++++++++++++++ mm/mm_init.c | 10 ++ mm/vmstat.c | 13 ++ 8 files changed, 407 insertions(+)create mode 100644
include/linux/kpromoted.h
create mode 100644mm/kpromoted.cinclude/linux/kpromoted.h new file +15
-0@@ -0,0 +1,54 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_KPROMOTED_H +#define _LINUX_KPROMOTED_H + +#include <linux/types.h> +#include <linux/init.h> +#include <linux/workqueue_types.h> + +/* Page hotness temperature sources */ +enum kpromoted_src { +KPROMOTED_HW_HINTS , +KPROMOTED_PGTABLE_SCAN , +}; + +#ifdef CONFIG_KPROMOTED
Quoted 8 lines
include/linux/kpromoted.h line 21
Quoted 6 lines
include/linux/kpromoted.h new file +6
-0+ +#define KPROMOTED_FREQ_WINDOW (5 * MSEC_PER_SEC) + +/* 2 accesses within a window will make the page a promotion candidate */ +#define KPRMOTED_FREQ_THRESHOLD 2 +Were these value derived empirically?
It is something I started with capture the notion of "repeated access".
Quoted 124 lines
Quoted 124 lines
include/linux/kpromoted.h new file +33
-0+#define KPROMOTED_HASH_ORDER 16 + +struct page_hotness_info { +unsigned long pfn; + +/* Time when this record was updated last */ +unsigned long last_update; + +/* +* Number of times this page was accessed in the +* current window +*/ +int frequency; + +/* Most recent access time */ +unsigned long recency; + +/* Most recent access from this node */ +int hot_node; +struct hlist_node hnode; +}; + +#define KPROMOTE_DELAY MSEC_PER_SEC + +int kpromoted_record_access (u64 pfn,int nid,int src,unsigned long now); +#else +static inline int kpromoted_record_access (u64 pfn,int nid,int src, +unsigned long now) +{ +return 0 ; +} +#endif /* CONFIG_KPROMOTED */ +#endif /* _LINUX_KPROMOTED_H */ include/linux/mmzone.h +4
-0@@ -1459,6 +1459,10 @@ typedef struct pglist_data {#ifdef CONFIG_MEMORY_FAILURE struct memory_failure_stats mf_stats;#endif +#ifdef CONFIG_KPROMOTED +struct task_struct *kpromoted; +wait_queue_head_t kpromoted_wait; +#endif } pg_data_t;#define node_present_pages (nid) (NODE_DATA(nid)->node_present_pages)include/linux/vm_event_item.h +13
-0@@ -182,6 +182,19 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, KSTACK_REST,#endif #endif /* CONFIG_DEBUG_STACK_USAGE */ +KPROMOTED_RECORDED_ACCESSES , +KPROMOTED_RECORD_HWHINTS , +KPROMOTED_RECORD_PGTSCANS , +KPROMOTED_RECORD_TOPTIER , +KPROMOTED_RECORD_ADDED , +KPROMOTED_RECORD_EXISTS , +KPROMOTED_MIG_RIGHT_NODE , +KPROMOTED_MIG_NON_LRU , +KPROMOTED_MIG_COLD_OLD , +KPROMOTED_MIG_COLD_NOT_ACCESSED , +KPROMOTED_MIG_CANDIDATE , +KPROMOTED_MIG_PROMOTED , +KPROMOTED_MIG_DROPPED ,NR_VM_EVENT_ITEMS };mm/Kconfig +7
-0@@ -1358,6 +1358,13 @@ config PT_RECLAIM Note: now only empty user PTE page table pages will be reclaimed. +config KPROMOTED + bool "Kernel hot page promotion daemon" + def_bool y + depends on NUMA && MIGRATION && MMU + help + Promote hot pages from lower tier to top tier by using the + memory access information provided by various sources. source "mm/damon/Kconfig"mm/Makefile +1
-0@@ -147,3 +147,4 @@ obj-$(CONFIG_SHRINKER_DEBUG) += shrinker_debug.o obj-$(CONFIG_EXECMEM) += execmem.o obj-$(CONFIG_TMPFS_QUOTA) += shmem_quota.o obj-$(CONFIG_PT_RECLAIM) += pt_reclaim.o +obj-$(CONFIG_KPROMOTED) += kpromoted.omm/kpromoted.c new file +17
-0@@ -0,0 +1,305 @@ +// SPDX-License-Identifier: GPL-2.0 +/* +* kpromoted is a kernel thread that runs on each node that has CPU i,e., +* on regular nodes. +* +* Maintains list of hot pages from lower tiers and promotes them. +*/ +#include <linux/kpromoted.h> +#include <linux/kthread.h> +#include <linux/mutex.h> +#include <linux/mmzone.h> +#include <linux/migrate.h> +#include <linux/memory-tiers.h> +#include <linux/slab.h> +#include <linux/sched.h> +#include <linux/cpuhotplug.h> +#include <linux/hashtable.h>
Quoted 8 lines
mm/kpromoted.c line 23
Quoted 6 lines
mm/kpromoted.c new file +6
-0+ +static DEFINE_HASHTABLE(page_hotness_hash, KPROMOTED_HASH_ORDER); +static struct mutex page_hotness_lock[1UL <<KPROMOTED_HASH_ORDER ]; + +static int kpromote_page (struct page_hotness_info *phi) +{Why not just call it
kpromote_folio?
Yes, can be called so.
Quoted 9 lines
mm/kpromoted.c line 29
Quoted 7 lines
mm/kpromoted.c new file +6
-0+struct page *page =pfn_to_page (phi->pfn); +struct folio *folio; +int ret; + +if (!page) +return 1 ;Do we need to check for
is_zone_device_page()here?
That and other checks are part of page_should_be_promoted() call just prior to attempting to promote.
Quoted 13 lines
mm/kpromoted.c line 37
Quoted 9 lines
mm/kpromoted.c new file +8
-0+ + folio =page_folio (page); + ret =migrate_misplaced_folio_prepare (folio,NULL , phi->hot_node); +if (ret) +return 1 ; + +return migrate_misplaced_folio (folio, phi->hot_node); +}Could you please document the assumptions for
kpromote_page(), what locks should be held? Does the ref count need to be incremented?
Yes, will document. However it doesn't expect folio refcount to be incremented as I am tracking hotpages via PFNs and not by using struct folios.
Quoted 18 lines
Quoted 18 lines
mm/kpromoted.c new file +17
-0+ +static int page_should_be_promoted (struct page_hotness_info *phi) +{ +struct page *page =pfn_to_online_page (phi->pfn); +unsigned long now = jiffies; +struct folio *folio; + +if (!page ||is_zone_device_page (page)) +return false; + + folio =page_folio (page); +if (!folio_test_lru (folio)) { +count_vm_event (KPROMOTED_MIG_NON_LRU ); +return false; + } +if (folio_nid (folio) == phi->hot_node) { +count_vm_event (KPROMOTED_MIG_RIGHT_NODE );
Quoted 8 lines
mm/kpromoted.c line 60
Quoted 6 lines
mm/kpromoted.c new file +6
-0+return false; + } + +/* If the page was hot a while ago, don't promote */ +if ((now - phi->last_update) >2 *msecs_to_jiffies (KPROMOTED_FREQ_WINDOW )) { +count_vm_event (KPROMOTED_MIG_COLD_OLD );Shouldn't we update
phi->last_updatehere?
Hmm I am not sure about updating from here where we are checking for migration feasibility. last_update records the time when the page was last accesed.
Quoted 43 lines
Quoted 43 lines
mm/kpromoted.c new file +42
-0+return false; + } + +/* If the page hasn't been accessed enough number of times, don't promote */ +if (phi->frequency <KPRMOTED_FREQ_THRESHOLD ) { +count_vm_event (KPROMOTED_MIG_COLD_NOT_ACCESSED ); +return false; + } +return true; +} + +/* +* Go thro' page hotness information and migrate pages if required. +* +* Promoted pages are not longer tracked in the hot list. +* Cold pages are pruned from the list as well. +* +* TODO: Batching could be done +*/ +static void kpromoted_migrate (pg_data_t *pgdat) +{ +int nid = pgdat->node_id; +struct page_hotness_info *phi; +struct hlist_node *tmp; +int nr_bkts =HASH_SIZE (page_hotness_hash); +int bkt; + +for (bkt =0 ; bkt < nr_bkts; bkt++) { +mutex_lock (&page_hotness_lock[bkt]); +hlist_for_each_entry_safe (phi, tmp, &page_hotness_hash[bkt], hnode) { +if (phi->hot_node != nid) +continue ; + +if (page_should_be_promoted (phi)) { +count_vm_event (KPROMOTED_MIG_CANDIDATE ); +if (!kpromote_page (phi)) { +count_vm_event (KPROMOTED_MIG_PROMOTED ); +hlist_del_init (&phi->hnode); +kfree (phi); + } + }else { +/*
Quoted 8 lines
mm/kpromoted.c line 108
Quoted 6 lines
mm/kpromoted.c new file +6
-0+* Not a suitable page or cold page, stop tracking it. +* TODO: Identify cold pages and drive demotion? +*/ +count_vm_event (KPROMOTED_MIG_DROPPED ); +hlist_del_init (&phi->hnode); +kfree (phi);Won't existing demotion already handle this?
Yes it does. I had a note here to check if it makes sense to drive demotion of pages that are being dropped off from kpromoted tracking presumably becasue they aren't hot any longer.
Quoted 10 lines
mm/kpromoted.c line 114
Quoted 7 lines
mm/kpromoted.c new file +6
-0+ } + } +mutex_unlock (&page_hotness_lock[bkt]); + } +} +It sounds like NUMA balancing, promotion and demotion can all act on parallel on these folios, if not could you clarify their relationship and dependency?
kpromoted tracks the hotness of PFNs. It goes through same steps that others use to isolate the pages prior to migration. So it is not possible to find a page that kpromoted wants to migrate being parallely considered by NUMAB for migration or vmscan for demotion. I don't see any obvious dependency here, but I can check in detail.
Quoted 97 lines
Quoted 97 lines
mm/kpromoted.c new file +95
-0+static struct page_hotness_info *__kpromoted_lookup (unsigned long pfn,int bkt) +{ +struct page_hotness_info *phi; + +hlist_for_each_entry (phi, &page_hotness_hash[bkt], hnode) { +if (phi->pfn == pfn) +return phi; + } +return NULL ; +} + +static struct page_hotness_info *kpromoted_lookup (unsigned long pfn,int bkt,unsigned long now) +{ +struct page_hotness_info *phi; + + phi =__kpromoted_lookup (pfn, bkt); +if (!phi) { + phi =kzalloc (sizeof (struct page_hotness_info ),GFP_KERNEL ); +if (!phi) +return ERR_PTR (-ENOMEM ); + + phi->pfn = pfn; + phi->frequency =1 ; + phi->last_update = now; + phi->recency = now; +hlist_add_head (&phi->hnode, &page_hotness_hash[bkt]); +count_vm_event (KPROMOTED_RECORD_ADDED ); + }else { +count_vm_event (KPROMOTED_RECORD_EXISTS ); + } +return phi; +} + +/* +* Called by subsystems that generate page hotness/access information. +* +* Records the memory access info for futher action by kpromoted. +*/ +int kpromoted_record_access (u64 pfn,int nid,int src,unsigned long now) +{ +struct page_hotness_info *phi; +struct page *page; +struct folio *folio; +int ret, bkt; + +count_vm_event (KPROMOTED_RECORDED_ACCESSES ); + +switch (src) { +case KPROMOTED_HW_HINTS : +count_vm_event (KPROMOTED_RECORD_HWHINTS ); +break ; +case KPROMOTED_PGTABLE_SCAN : +count_vm_event (KPROMOTED_RECORD_PGTSCANS ); +break ; +default : +break ; + } + +/* +* Record only accesses from lower tiers. +* Assuming node having CPUs as toptier for now. +*/ +if (node_is_toptier (pfn_to_nid (pfn))) { +count_vm_event (KPROMOTED_RECORD_TOPTIER ); +return 0 ; + } + + page =pfn_to_online_page (pfn); +if (!page ||is_zone_device_page (page)) +return 0 ; + + folio =page_folio (page); +if (!folio_test_lru (folio)) +return 0 ; + + bkt =hash_min (pfn,KPROMOTED_HASH_ORDER ); +mutex_lock (&page_hotness_lock[bkt]); + phi =kpromoted_lookup (pfn, bkt, now); +if (!phi) { + ret =PTR_ERR (phi); + goto out; + } + +if ((phi->last_update - now) >msecs_to_jiffies (KPROMOTED_FREQ_WINDOW )) { +/* New window */ + phi->frequency =1 ;/* TODO: Factor in the history */ + phi->last_update = now; + }else { + phi->frequency++; + } + phi->recency = now; + +/* +* TODOs: +* 1. Source nid is hard-coded for some temperature sources
Quoted 8 lines
mm/kpromoted.c line 213
Quoted 4 lines
mm/kpromoted.c new file +4
-0+* 2. Take action if hot_node changes - may be a shared page? +* 3. Maintain node info for every access within the window? +*/ + phi->hot_node = (nid ==NUMA_NO_NODE ) ?1 : nid;I don't understand why nid needs to be 1 if nid is
NUMA_NODE_ID? Does it mean that it's being promoted to the top tier, the mix ofhot_node, tier and nid is not very clear here.
As I mentioned earlier, if the access information wasn't accompanied by nid (which is specified by NUMA_NO_NODE), it will be promoted to a hard-code (currently) toptier node.
Quoted 84 lines
Quoted 84 lines
mm/kpromoted.c new file +83
-0+mutex_unlock (&page_hotness_lock[bkt]); +out: +return 0 ; +} + +/* +* Go through the accumulated mem_access_info and migrate +* pages if required. +*/ +static void kpromoted_do_work (pg_data_t *pgdat) +{ +kpromoted_migrate (pgdat); +} + +static inline bool kpromoted_work_requested (pg_data_t *pgdat) +{ +return false; +} + +static int kpromoted (void *p) +{ +pg_data_t *pgdat = (pg_data_t *)p; +struct task_struct *tsk = current; +long timeout =msecs_to_jiffies (KPROMOTE_DELAY ); + +const struct cpumask *cpumask =cpumask_of_node (pgdat->node_id); + +if (!cpumask_empty (cpumask)) +set_cpus_allowed_ptr (tsk, cpumask); + +while (!kthread_should_stop ()) { +wait_event_timeout (pgdat->kpromoted_wait, +kpromoted_work_requested (pgdat), timeout); +kpromoted_do_work (pgdat); + } +return 0 ; +} + +static void kpromoted_run (int nid) +{ +pg_data_t *pgdat =NODE_DATA (nid); + +if (pgdat->kpromoted) +return ; + + pgdat->kpromoted =kthread_run (kpromoted, pgdat,"kpromoted%d" , nid); +if (IS_ERR (pgdat->kpromoted)) { +pr_err ("Failed to start kpromoted on node %d\n" , nid); + pgdat->kpromoted =NULL ; + } +} + +static int kpromoted_cpu_online (unsigned int cpu) +{ +int nid; + +for_each_node_state (nid,N_CPU ) { +pg_data_t *pgdat =NODE_DATA (nid); +const struct cpumask *mask; + + mask =cpumask_of_node (pgdat->node_id); + +if (cpumask_any_and (cpu_online_mask, mask) < nr_cpu_ids) +/* One of our CPUs online: restore mask */ +if (pgdat->kpromoted) +set_cpus_allowed_ptr (pgdat->kpromoted, mask); + } +return 0 ; +} + +static int __init kpromoted_init (void ) +{ +int nid, ret, i; + + ret =cpuhp_setup_state_nocalls (CPUHP_AP_ONLINE_DYN , +"mm/promotion:online" , + kpromoted_cpu_online,NULL ); +if (ret <0 ) { +pr_err ("kpromoted: failed to register hotplug callbacks.\n" ); +return ret; + } + +for (i =0 ; i < (1UL <<KPROMOTED_HASH_ORDER ); i++)
Quoted 8 lines
mm/kpromoted.c line 301
Quoted 5 lines
mm/kpromoted.c new file +5
-0+mutex_init (&page_hotness_lock[i]); + +for_each_node_state (nid,N_CPU ) +kpromoted_run (nid); +I think we need a dynamic way to disabling promotion at run time as well, right?
Myabe, but I understand that promotion is an activity that should be benefitial in general. What specific scenarios do you think would need explicit disabling of promotion?
Regards,
Bharata.
Gregory Price
reviewerRe: [RFC PATCH 2/4] mm: kpromoted: Hot page info collection and promotion daemon
In reply to Bharata B RaoQuoted 5 lines · Bharata B Rao
kpromoted is a kernel daemon that accumulates hot page info from different sources and tries to promote pages from slow tiers to top tiers. One instance of this thread runs on each node that has CPUs.
Hot take: This sounds more like ktieringd not kpromoted
Is it reasonable to split the tracking a promotion logic into separate interfaces? This would let us manage, for example, rate-limiting in the movement interface cleanly without having to care about the tiering system(s) associated with it.
my_tiering_magic():
... identify hot things ...
promote(batch_folios, optional_data);
-> kick daemon thread to wake up and do the promotion
... continue async things ...Optional data could be anything from target nodes or accessor info, but not hotness information.
Then users at least get a clean interface for things like rate-limiting, and everyone proposing their own take on tiering can consume it. This may also be useful for existing users (TPP, reclaim?, etc).
~Gregory
Re: [RFC PATCH 2/4] mm: kpromoted: Hot page info collection and promotion daemon
Quoted 8 lines · Gregory Price
Quoted 5 lines · Bharata B Rao
kpromoted is a kernel daemon that accumulates hot page info from different sources and tries to promote pages from slow tiers to top tiers. One instance of this thread runs on each node that has CPUs.
Hot take: This sounds more like ktieringd not kpromoted
:-)
Quoted 12 lines
Is it reasonable to split the tracking a promotion logic into separate interfaces? This would let us manage, for example, rate-limiting in the movement interface cleanly without having to care about the tiering system(s) associated with it.
my_tiering_magic(): ... identify hot things ... promote(batch_folios, optional_data); -> kick daemon thread to wake up and do the promotion ... continue async things ...
Quoted 6 lines
Optional data could be anything from target nodes or accessor info, but not hotness information.
Then users at least get a clean interface for things like rate-limiting, and everyone proposing their own take on tiering can consume it. This may also be useful for existing users (TPP, reclaim?, etc).
Yes, Makes sense to split tracking and promotion logic into separate parts. There is no need for the promotion part to work with the hot page list that belongs to the tracking part as I have done in this RFC.
Raghu and I already saw that migration part is kind of duplicated in our patchsets(kmmscand and this) and were thinking of unifying them. Having clean separation as you suggest will be good.
Regards,
Bharata.
RFC PATCH 3/4 x86: ibs: In-kernel IBS driver for memory access profiling
In reply to Bharata B RaoUse IBS (Instruction Based Sampling) feature present in AMD processors for memory access tracking. The access information obtained from IBS via NMI is fed to kpromoted daemon for futher action.
In addition to many other information related to the memory access, IBS provides physical (and virtual) address of the access and indicates if the access came from slower tier. Only memory accesses originating from slower tiers are further acted upon by this driver.
The samples are initially accumulated in percpu buffers which are flushed to kpromoted using irq_work.
About IBS
---------
IBS can be programmed to provide data about instruction execution periodically. This is done by programming a desired sample count (number of ops) in a control register. When the programmed number of ops are dispatched, a micro-op gets tagged, various information about the tagged micro-op's execution is populated in IBS execution MSRs and an interrupt is raised. While IBS provides a lot of data for each sample, for the purpose of memory access profiling, we are interested in linear and physical address of the memory access that reached DRAM. Recent AMD processors provide further filtering where it is possible to limit the sampling to those ops that had an L3 miss which greately reduces the non-useful samples.
While IBS provides capability to sample instruction fetch and execution, only IBS execution sampling is used here to collect data about memory accesses that occur during the instruction execution.
More information about IBS is available in Sec 13.3 of AMD64 Architecture Programmer's Manual, Volume 2:System Programming which is present at:
https://bugzilla.kernel.org/attachment.cgi?id=288923
Information about MSRs used for programming IBS can be found in Sec 2.1.14.4 of PPR Vol 1 for AMD Family 19h Model 11h B1 which is currently present at:
https://www.amd.com/system/files/TechDocs/55901_0.25.zip
--- arch/x86/events/amd/ibs.c | 11 ++ arch/x86/include/asm/ibs.h | 7 + arch/x86/include/asm/msr-index.h | 16 ++ arch/x86/mm/Makefile | 3 +- arch/x86/mm/ibs.c | 312 +++++++++++++++++++++++++++++++ include/linux/vm_event_item.h | 17 ++ mm/vmstat.c | 17 ++ 7 files changed, 382 insertions(+), 1 deletion(-) create mode 100644 arch/x86/include/asm/ibs.h create mode 100644 arch/x86/mm/ibs.c
arch/x86/events/amd/ibs.c +11 -0
@@ -13,8 +13,10 @@#include <linux/ptrace.h> #include <linux/syscore_ops.h> #include <linux/sched/clock.h> +#include <linux/kpromoted.h> #include <asm/apic.h> +#include <asm/ibs.h> #include "../perf_event.h"
@@ -1539,6 +1541,15 @@ static __init int amd_ibs_init(void) {u32 caps; +/* +* TODO: Find a clean way to disable perf IBS so that IBS +* can be used for memory access profiling. +*/ +if (arch_hw_access_profiling) { +pr_info ("IBS isn't available for perf use\n" ); +return 0 ; + } + caps =__get_ibs_caps ();if (!caps)return -ENODEV ;/* ibs not supported by the cpu */
arch/x86/include/asm/ibs.h new file +7 -0
@@ -0,0 +1,7 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_X86_IBS_H +#define _ASM_X86_IBS_H + +extern bool arch_hw_access_profiling; + +#endif /* _ASM_X86_IBS_H */
arch/x86/include/asm/msr-index.h +16 -0
@@ -719,6 +719,22 @@/* AMD Last Branch Record MSRs */ #define MSR_AMD64_LBR_SELECT 0xc000010e +/* AMD IBS MSR bits */ +#define MSR_AMD64_IBSOPDATA2_DATASRC 0x7 +#define MSR_AMD64_IBSOPDATA2_DATASRC_LCL_CACHE 0x1 +#define MSR_AMD64_IBSOPDATA2_DATASRC_PEER_CACHE_NEAR 0x2 +#define MSR_AMD64_IBSOPDATA2_DATASRC_DRAM 0x3 +#define MSR_AMD64_IBSOPDATA2_DATASRC_FAR_CCX_CACHE 0x5 +#define MSR_AMD64_IBSOPDATA2_DATASRC_EXT_MEM 0x8 +#define MSR_AMD64_IBSOPDATA2_RMTNODE 0x10 + +#define MSR_AMD64_IBSOPDATA3_LDOP BIT_ULL(0) +#define MSR_AMD64_IBSOPDATA3_STOP BIT_ULL(1) +#define MSR_AMD64_IBSOPDATA3_DCMISS BIT_ULL(7) +#define MSR_AMD64_IBSOPDATA3_LADDR_VALID BIT_ULL(17) +#define MSR_AMD64_IBSOPDATA3_PADDR_VALID BIT_ULL(18) +#define MSR_AMD64_IBSOPDATA3_L2MISS BIT_ULL(20) +/* Zen4 */ #define MSR_ZEN4_BP_CFG 0xc001102e#define MSR_ZEN4_BP_CFG_SHARED_BTB_FIX_BIT 5
arch/x86/mm/Makefile +2 -1
@@ -26,7 +26,8 @@ CFLAGS_REMOVE_pgprot.o = -pg endif obj-y := init.o init_$(BITS).o fault.o ioremap.o extable.o mmap.o \ - pgtable.o physaddr.o tlb.o cpu_entry_area.o maccess.o pgprot.o + pgtable.o physaddr.o tlb.o cpu_entry_area.o maccess.o pgprot.o \ + ibs.o obj-y += pat/
arch/x86/mm/ibs.c new file +312 -0
312 lines not on this page · raw message
include/linux/vm_event_item.h +17 -0
@@ -195,6 +195,23 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,KPROMOTED_MIG_CANDIDATE ,KPROMOTED_MIG_PROMOTED ,KPROMOTED_MIG_DROPPED , +HWHINT_NR_EVENTS , +HWHINT_KERNEL , +HWHINT_KTHREAD , +HWHINT_NON_LOAD_STORES , +HWHINT_DC_L2_HITS , +HWHINT_LOCAL_L3L1L2 , +HWHINT_LOCAL_PEER_CACHE_NEAR , +HWHINT_FAR_CACHE_HITS , +HWHINT_DRAM_ACCESSES , +HWHINT_CXL_ACCESSES , +HWHINT_REMOTE_NODE , +HWHINT_LADDR_INVALID , +HWHINT_KERNEL_ADDR , +HWHINT_PADDR_INVALID , +HWHINT_NON_LRU , +HWHINT_BUFFER_FULL , +HWHINT_USEFUL_SAMPLES ,NR_VM_EVENT_ITEMS };
mm/vmstat.c +17 -0
@@ -1479,6 +1479,23 @@ const char * const vmstat_text[] = {"kpromoted_mig_candidate" ,"kpromoted_mig_promoted" ,"kpromoted_mig_dropped" , +"hwhint_nr_events" , +"hwhint_kernel" , +"hwhint_kthread" , +"hwhint_non_load_stores" , +"hwhint_dc_l2_hits" , +"hwhint_local_l3l1l2" , +"hwhint_local_peer_cache_near" , +"hwhint_far_cache_hits" , +"hwhint_dram_accesses" , +"hwhint_cxl_accesses" , +"hwhint_remote_node" , +"hwhint_invalid_laddr" , +"hwhint_kernel_addr" , +"hwhint_invalid_paddr" , +"hwhint_non_lru" , +"hwhint_buffer_full" , + "hwhint_useful_samples", #endif /* CONFIG_VM_EVENT_COUNTERS || CONFIG_MEMCG */ };#endif /* CONFIG_PROC_FS || CONFIG_SYSFS || CONFIG_NUMA || CONFIG_MEMCG */
Signature · 1 line
2.34.1
Re: [RFC PATCH 3/4] x86: ibs: In-kernel IBS driver for memory access profiling
Quoted 39 lines · On Thu, 6 Mar 2025 11:15:31 +0530 Bharata B Rao <[email protected]> wrote:
Use IBS (Instruction Based Sampling) feature present in AMD processors for memory access tracking. The access information obtained from IBS via NMI is fed to kpromoted daemon for futher action.
In addition to many other information related to the memory access, IBS provides physical (and virtual) address of the access and indicates if the access came from slower tier. Only memory accesses originating from slower tiers are further acted upon by this driver.
The samples are initially accumulated in percpu buffers which are flushed to kpromoted using
irq_work.About IBS
---------
IBS can be programmed to provide data about instruction execution periodically. This is done by programming a desired sample count (number of ops) in a control register. When the programmed number of ops are dispatched, a micro-op gets tagged, various information about the tagged micro-op's execution is populated in IBS execution MSRs and an interrupt is raised. While IBS provides a lot of data for each sample, for the purpose of memory access profiling, we are interested in linear and physical address of the memory access that reached DRAM. Recent AMD processors provide further filtering where it is possible to limit the sampling to those ops that had an L3 miss which greately reduces the non-useful samples.While IBS provides capability to sample instruction fetch and execution, only IBS execution sampling is used here to collect data about memory accesses that occur during the instruction execution.
More information about IBS is available in Sec 13.3 of AMD64 Architecture Programmer's Manual, Volume 2:System Programming which is present at:
https://bugzilla.kernel.org/attachment.cgi?id=288923
Quoted 7 lines
Information about MSRs used for programming IBS can be found in Sec 2.1.14.4 of PPR Vol 1 for AMD Family 19h Model 11h B1 which is currently present at:
https://www.amd.com/system/files/TechDocs/55901_0.25.zipSigned-off-by Bharata B Rao <[email protected]>---
Trivial comments inline. I'd love to find a clean way to steal stuff perf is using though.
Quoted 27 lines
arch/x86/events/amd/ibs.c | 11 ++ arch/x86/include/asm/ibs.h | 7 + arch/x86/include/asm/msr-index.h | 16 ++ arch/x86/mm/Makefile | 3 +- arch/x86/mm/ibs.c | 312 +++++++++++++++++++++++++++++++ include/linux/vm_event_item.h | 17 ++ mm/vmstat.c | 17 ++ 7 files changed, 382 insertions(+), 1 deletion(-) create mode 100644 arch/x86/include/asm/ibs.h create mode 100644 arch/x86/mm/ibs.carch/x86/events/amd/ibs.c +2
-0@@ -13,8 +13,10 @@#include <linux/ptrace.h> #include <linux/syscore_ops.h> #include <linux/sched/clock.h> +#include <linux/kpromoted.h> #include <asm/apic.h> +#include <asm/ibs.h> #include "../perf_event.h"
arch/x86/events/amd/ibs.c static __init int amd_ibs_init(void) · line 1546
Quoted 6 lines
arch/x86/events/amd/ibs.c +3
-0@@ -1539,6 +1541,15 @@ static __init int amd_ibs_init(void) {u32 caps; +/* +* TODO: Find a clean way to disable perf IBS so that IBS +* can be used for memory access profiling.
Yeah. That bit us in a number of similar cases. Does anyone have a good solution for this? For my hammer (CXL HMU) the perf case is probably the niche one so I'm less worried, but for SPE, IBS, PEBS etc we need to figure out how to elegantly back off on promotion if a user wants to use tracing.
Quoted 7 lines
arch/x86/events/amd/ibs.c +6
-0+*/ +if (arch_hw_access_profiling) { +pr_info ("IBS isn't available for perf use\n" ); +return 0 ; + } + caps =__get_ibs_caps ();
arch/x86/events/amd/ibs.c static __init int amd_ibs_init(void)
Quoted 10 lines
arch/x86/events/amd/ibs.c +6
-0if (!caps)return -ENODEV ;/* ibs not supported by the cpu */ + +static void clear_APIC_ibs (void ) +{ +int offset; + + offset =get_ibs_lvt_offset ();
Trivial but I'd flip condition and deal with the error out of line. Ah I see this is cut and paste from existing code I'll stop pointing this stuff out!
if (offset < 0) return; setup_APIC_eivt();
arch/x86/events/amd/ibs.c
Quoted 9 lines
arch/x86/events/amd/ibs.c +8
-0+if (offset >=0 ) +setup_APIC_eilvt (offset,0 ,APIC_EILVT_MSG_FIX ,1 ); +} + +static int __init ibs_access_profiling_init (void ) +{ +if (!boot_cpu_has (X86_FEATURE_IBS )) { +pr_info ("IBS capability is unavailable for access profiling\n" );
Probably worth saying that is because the chip doesn't have it! This reads to similar to the perf case above where we just pinched it for other usecases.
Quoted 28 lines
arch/x86/events/amd/ibs.c +28
-0+return 0 ; + } + + ibs_s = alloc_percpu_gfp(struct ibs_sample_pcpu, __GFP_ZERO); +if (!ibs_s) +return 0 ; + +INIT_WORK (&ibs_work, ibs_work_handler); +init_irq_work (&ibs_irq_work, ibs_irq_handler); + +/* Uses IBS Op sampling */ + ibs_config =IBS_OP_CNT_CTL |IBS_OP_ENABLE ; + ibs_caps =cpuid_eax (IBS_CPUID_FEATURES ); +if (ibs_caps &IBS_CAPS_ZEN4 ) + ibs_config |=IBS_OP_L3MISSONLY ; + +register_nmi_handler (NMI_LOCAL , ibs_overflow_handler,0 ,"ibs" ); + +cpuhp_setup_state (CPUHP_AP_PERF_X86_AMD_IBS_STARTING , +"x86/amd/ibs_access_profile:starting" , + x86_amd_ibs_access_profile_startup, + x86_amd_ibs_access_profile_teardown); + +pr_info ("IBS setup for memory access profiling\n" ); +return 0 ; +} + +arch_initcall (ibs_access_profiling_init);
RFC PATCH 4/4 x86: ibs: Enable IBS profiling for memory accesses
In reply to Bharata B RaoEnable IBS memory access data collection for user memory accesses by programming the required MSRs. The profiling is turned ON only for user mode execution and turned OFF for kernel mode execution. Profiling is explicitly disabled for NMI handler too.
TODOs:
- IBS sampling rate is kept fixed for now.
- Arch/vendor separation/isolation of the code needs relook.
--- arch/x86/include/asm/entry-common.h | 3 +++ arch/x86/include/asm/hardirq.h | 2 ++ arch/x86/include/asm/ibs.h | 2 ++ arch/x86/mm/ibs.c | 32 +++++++++++++++++++++++++++++ 4 files changed, 39 insertions(+)
arch/x86/include/asm/entry-common.h +3 -0
@@ -9,10 +9,12 @@#include <asm/io_bitmap.h> #include <asm/fpu/api.h> #include <asm/fred.h> +#include <asm/ibs.h> /* Check that the stack and regs on entry from user mode are sane. */ static __always_inline void arch_enter_from_user_mode(struct pt_regs *regs) { +hw_access_profiling_stop ();if (IS_ENABLED (CONFIG_DEBUG_ENTRY )) {/* * Make sure that the entry code gave us a sensible EFLAGS
@@ -98,6 +100,7 @@ static inline void arch_exit_to_user_mode_prepare(struct pt_regs *regs, static __always_inline void arch_exit_to_user_mode(void) {amd_clear_divider (); +hw_access_profiling_start (); }#define arch_exit_to_user_mode arch_exit_to_user_mode
arch/x86/include/asm/hardirq.h +2 -0
@@ -91,4 +91,6 @@ static __always_inline bool kvm_get_cpu_l1tf_flush_l1d(void) static __always_inline void kvm_set_cpu_l1tf_flush_l1d(void) { }#endif /* IS_ENABLED(CONFIG_KVM_INTEL) */ +#define arch_nmi_enter () hw_access_profiling_stop() +#define arch_nmi_exit () hw_access_profiling_start()#endif /* _ASM_X86_HARDIRQ_H */
arch/x86/include/asm/ibs.h +2 -0
@@ -2,6 +2,8 @@#ifndef _ASM_X86_IBS_H#define _ASM_X86_IBS_H +void hw_access_profiling_start (void ); +void hw_access_profiling_stop (void );extern bool arch_hw_access_profiling;#endif /* _ASM_X86_IBS_H */
arch/x86/mm/ibs.c +32 -0
@@ -15,6 +15,7 @@ bool arch_hw_access_profiling;static u64 ibs_config__read_mostly ;static u32 ibs_caps; +#define IBS_SAMPLE_PERIOD 10000#define IBS_NR_SAMPLES 50/*
@@ -99,6 +100,36 @@ static void ibs_irq_handler(struct irq_work *i)schedule_work_on (smp_processor_id (), &ibs_work); } +void hw_access_profiling_stop (void ) +{ +u64 ops_ctl; + +if (!arch_hw_access_profiling) +return ; + +rdmsrl (MSR_AMD64_IBSOPCTL , ops_ctl); +wrmsrl (MSR_AMD64_IBSOPCTL , ops_ctl & ~IBS_OP_ENABLE ); +} + +void hw_access_profiling_start (void ) +{ +u64 config =0 ; +unsigned int period =IBS_SAMPLE_PERIOD ; + +if (!arch_hw_access_profiling) +return ; + +/* Disable IBS for kernel thread */ +if (!current->mm) + goto out; + + config = (period >>4 ) &IBS_OP_MAX_CNT ; + config |= (period &IBS_OP_MAX_CNT_EXT_MASK ); + config |= ibs_config; +out: +wrmsrl (MSR_AMD64_IBSOPCTL , config); +} +/* * IBS NMI handler: Process the memory access info reported by IBS. *
@@ -305,6 +336,7 @@ static int __init ibs_access_profiling_init(void) x86_amd_ibs_access_profile_startup, x86_amd_ibs_access_profile_teardown); + arch_hw_access_profiling = true;pr_info ("IBS setup for memory access profiling\n" );return 0 ; }
Signature · 1 line
2.34.1
kernel test robot
build botRe: [RFC PATCH 4/4] x86: ibs: Enable IBS profiling for memory accesses
kernel test robot noticed the following build warnings:
Hi Bharata,
[This is a private test report for your RFC patch.] kernel test robot noticed the following build warnings:
[auto build test WARNING on tip/x86/core]
[also build test WARNING on tip/x86/mm linus/master v6.14-rc5] [cannot apply to akpm-mm/mm-everything next-20250306]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Bharata-B-Rao/mm-migrate-Allow-misplaced-migration-without-VMA-too/20250306-134949 base: tip/x86/core patch link: https://lore.kernel.org/r/20250306054532.221138-5-bharata%40amd.com patch subject: [RFC PATCH 4/4] x86: ibs: Enable IBS profiling for memory accesses config: x86_64-buildonly-randconfig-003-20250307 (https://download.01.org/0day-ci/archive/20250307/[email protected]/config) compiler: clang version 19.1.7 (https://github.com/llvm/llvm-project cd708029e0b2869e80abe31ddb175f7c35361f90) reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250307/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All warnings (new ones prefixed by >>):
>> vmlinux.o: warning: objtool: do_syscall_64+0x13: call to hw_access_profiling_stop() leaves .noinstr.text section >> vmlinux.o: warning: objtool: syscall_enter_from_user_mode_prepare+0x5: call to hw_access_profiling_stop() leaves .noinstr.text section >> vmlinux.o: warning: objtool: syscall_exit_to_user_mode+0x7b: call to hw_access_profiling_start() leaves .noinstr.text section >> vmlinux.o: warning: objtool: irqentry_enter_from_user_mode+0x5: call to hw_access_profiling_stop() leaves .noinstr.text section >> vmlinux.o: warning: objtool: irqentry_exit_to_user_mode+0x71: call to hw_access_profiling_start() leaves .noinstr.text section >> vmlinux.o: warning: objtool: irqentry_nmi_enter+0x16: call to hw_access_profiling_stop() leaves .noinstr.text section >> vmlinux.o: warning: objtool: irqentry_nmi_exit+0x22: call to hw_access_profiling_start() leaves .noinstr.text section
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
kernel test robot
build botRe: [RFC PATCH 4/4] x86: ibs: Enable IBS profiling for memory accesses
In reply to Bharata B Raokernel test robot noticed the following build warnings:
Hi Bharata,
[This is a private test report for your RFC patch.] kernel test robot noticed the following build warnings:
[auto build test WARNING on tip/x86/core]
[also build test WARNING on tip/x86/mm linus/master v6.14-rc5] [cannot apply to akpm-mm/mm-everything next-20250307]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Bharata-B-Rao/mm-migrate-Allow-misplaced-migration-without-VMA-too/20250306-134949 base: tip/x86/core patch link: https://lore.kernel.org/r/20250306054532.221138-5-bharata%40amd.com patch subject: [RFC PATCH 4/4] x86: ibs: Enable IBS profiling for memory accesses config: x86_64-buildonly-randconfig-005-20250307 (https://download.01.org/0day-ci/archive/20250308/[email protected]/config) compiler: clang version 19.1.7 (https://github.com/llvm/llvm-project cd708029e0b2869e80abe31ddb175f7c35361f90) reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250308/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All warnings (new ones prefixed by >>):
vmlinux.o: warning: objtool: do_syscall_64+0x11: call to hw_access_profiling_stop() leaves .noinstr.text section
>> vmlinux.o: warning: objtool: do_int80_emulation+0x16: call to hw_access_profiling_stop() leaves .noinstr.text section
vmlinux.o: warning: objtool: syscall_enter_from_user_mode_prepare+0x5: call to hw_access_profiling_stop() leaves .noinstr.text section vmlinux.o: warning: objtool: syscall_exit_to_user_mode+0xcc: call to hw_access_profiling_start() leaves .noinstr.text section vmlinux.o: warning: objtool: irqentry_enter_from_user_mode+0x5: call to hw_access_profiling_stop() leaves .noinstr.text section vmlinux.o: warning: objtool: irqentry_exit_to_user_mode+0xc2: call to hw_access_profiling_start() leaves .noinstr.text section vmlinux.o: warning: objtool: irqentry_nmi_enter+0x1c: call to hw_access_profiling_stop() leaves .noinstr.text section vmlinux.o: warning: objtool: irqentry_nmi_exit+0x63: call to hw_access_profiling_start() leaves .noinstr.text section vmlinux.o: warning: objtool: ia32_setup_rt_frame+0x135: call to write_comp_data.llvm.15977173092665352742() with UACCESS enabled vmlinux.o: warning: objtool: compat_get_bitmap+0xb3: call to write_comp_data.llvm.15977173092665352742() with UACCESS enabled vmlinux.o: warning: objtool: compat_put_bitmap+0xb3: call to write_comp_data.llvm.15977173092665352742() with UACCESS enabled vmlinux.o: warning: objtool: __sanitizer_cov_trace_cmp1+0xd: call to write_comp_data.llvm.15977173092665352742() with UACCESS enabled vmlinux.o: warning: objtool: __sanitizer_cov_trace_cmp2+0x10: call to write_comp_data.llvm.15977173092665352742() with UACCESS enabled vmlinux.o: warning: objtool: __sanitizer_cov_trace_cmp4+0x10: call to write_comp_data.llvm.15977173092665352742() with UACCESS enabled vmlinux.o: warning: objtool: __sanitizer_cov_trace_cmp8+0xf: call to write_comp_data.llvm.15977173092665352742() with UACCESS enabled vmlinux.o: warning: objtool: __sanitizer_cov_trace_const_cmp1+0x10: call to write_comp_data.llvm.15977173092665352742() with UACCESS enabled vmlinux.o: warning: objtool: __sanitizer_cov_trace_const_cmp2+0x10: call to write_comp_data.llvm.15977173092665352742() with UACCESS enabled vmlinux.o: warning: objtool: __sanitizer_cov_trace_const_cmp4+0x10: call to write_comp_data.llvm.15977173092665352742() with UACCESS enabled vmlinux.o: warning: objtool: __sanitizer_cov_trace_const_cmp8+0xf: call to write_comp_data.llvm.15977173092665352742() with UACCESS enabled vmlinux.o: warning: objtool: __sanitizer_cov_trace_switch+0x51: call to write_comp_data.llvm.15977173092665352742() with UACCESS enabled vmlinux.o: warning: objtool: fault_in_writeable+0xae: call to write_comp_data.llvm.15977173092665352742() with UACCESS enabled vmlinux.o: warning: objtool: fault_in_readable+0xb5: call to write_comp_data.llvm.15977173092665352742() with UACCESS enabled vmlinux.o: warning: objtool: fillonedir+0x179: call to write_comp_data.llvm.15977173092665352742() with UACCESS enabled vmlinux.o: warning: objtool: filldir+0x245: call to write_comp_data.llvm.15977173092665352742() with UACCESS enabled vmlinux.o: warning: objtool: filldir64+0x244: call to write_comp_data.llvm.15977173092665352742() with UACCESS enabled vmlinux.o: warning: objtool: compat_fillonedir+0x1ae: call to write_comp_data.llvm.15977173092665352742() with UACCESS enabled vmlinux.o: warning: objtool: compat_filldir+0x28e: call to write_comp_data.llvm.15977173092665352742() with UACCESS enabled vmlinux.o: warning: objtool: copy_iovec_from_user+0x98: call to write_comp_data.llvm.15977173092665352742() with UACCESS enabled vmlinux.o: warning: objtool: check_zeroed_user+0xce: call to write_comp_data.llvm.15977173092665352742() with UACCESS enabled vmlinux.o: warning: objtool: strncpy_from_user+0x8a: call to write_comp_data.llvm.15977173092665352742() with UACCESS enabled vmlinux.o: warning: objtool: strnlen_user+0xa7: call to write_comp_data.llvm.15977173092665352742() with UACCESS enabled vmlinux.o: warning: objtool: copy_mc_fragile_handle_tail+0x4d: call to write_comp_data.llvm.15977173092665352742() with UACCESS enabled
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
+ Harry, who was called Hyeonggon before.
Hello,
Thank you very much for sharing this great patchset.
Quoted 5 lines · On Thu, 6 Mar 2025 11:15:28 +0530 Bharata B Rao <[email protected]> wrote:
Hi,
This is an attempt towards having a single subsystem that accumulates hot page information from lower memory tiers and does hot page promotion.
That is one of DAMON's goal, too. DAMON aims to be a kernel subsystem that can provide access information that accumulated from multiple sources and can be useful for multiple use cases including profiling and access aware system operations.
Hot pages information and promotioning those are one of such information and operations. SK hynix developed their CXL memory tiering solution[1] using DAMON. I also shared auto-tuning based memory tiering solution idea[2] before. On LSFMMBPF 2025, I may share its prototype implementation and evaluation results on CXL memory devices that I recentily gained access.
Of course, DAMON is still in the middle of its journey towards the northern star. I'm looking for what are really required to DAMON for the goal, what are [not] available with today's DAMON, and what should be the good future plans. My LSFMMBPF 2025 topic proposals are for those.
Hence, this patchset is very helpful to me at showing what can be added and improved on DAMON. I specifically understand support of access information sources other than Page tables' accessed bits such as AMD IBS as the main thing. I admit the fact that DAMON of today is supporting only page tables' accessed bit as the primary source of the information. But DAMON of future would be different. Let me share more thoughts below.
Quoted 7 lines
At the heart of this subsystem is a kernel daemon named kpromoted that does the following:
1. Exposes an API that other subsystems which detect/generate memory access information can use to inform the daemon about memory accesses from lower memory tiers.
DAMON also provides such API, namely, its monitoring operations set layer interface[3]. Nevertheless, only page tables accessed bit use cases exist today. Hence the interface may have have hidden problems at extending for other sources.
Quoted 2 lines
2. Maintains the list of hot pages and attempts to promote them to toptiers.
DAMON provides its another half, DAMOS[4], for this kind of usages.
Quoted 5 lines
Currently I have added AMD IBS driver as one source that provides page access information as an example. This driver feeds info to kpromoted in this RFC patchset. More sources were discussed in a similar context here at [1].
I was imagining how I would be able to do this with DAMON via operations set layer interface. And I find thee current interface is not very optimized for AMD IBS like sources that catches the access on the line. That is, in a way, we could say AMD IBS like primitives as push-oriented, while page tables' accessed bits information are pull-oriented. DAMON operations set layer interface is easier to be used in pull-oriented case. I don't think it cannot be used for push-oriented case, but definitely the interface would better to be more optimized for the use case.
I'm curious if you also tried doing this by extending DAMON, and if some hidden problems you found.
Quoted 6 lines
This is just an early attempt to check what it takes to maintain a single source of page hotness info and also separate hot page detection mechanisms from the promotion mechanism. There are too many open ends right now and I have listed a few of them below.
Quoted 10 lines
- The API that is provided to register memory access expects the PFN, NID and time of access at the minimum. This is described more in patch 2/4. This API currently can be called only from contexts that allow sleeping and hence this rules out using it from PTE scanning paths. The API needs to be more flexible with respect to this.
- Some sources like PTE A bit scanning can't provide the precise time of access or the NID that is accessing the page. The latter has been an open problem to which I haven't come across a good and acceptable solution.
Agree. PTE A bit scanning could be useful in many cases, but not every case. There was an RFC patchset[7] that extends DAMON for NID. I'm planning to do that again using DAMON operations layer interface. My current plan is to implement the prototype using prot_none page faults, and later extend for AMD IBS like h/w features. Hopefully I will share a prototype or at least more detailed idea on LSFMMBPF 2025.
Quoted 4 lines
- The way the hot page information is maintained is pretty primitive right now. Ideally we would like to store hotness info in such a way that it should be easily possible to lookup say N most hot pages.
DAMON provides a feature for lookup of N most hotpages, namely DAMOS quotas' access pattern based regions prioritization[5].
Quoted 4 lines
- If PTE A bit scanners are considered as hotness sources, we will be bombarded with accesses. Do we want to accomodate all those accesses or just go with hotness info for fixed number of pages (possibly as a ratio of lower tier memory capacity)?
I understand you're saying about memory space overhead. Correct me if I'm wrong, please.
Isn't same issue exists for current implementation of the sampling frequency is high, and/or aggregation window is long?
To me, hence, this looks like not a problem of the information source, but how to maintain the information. Current implementation maintains it per page, so I think the problem is inherent.
DAMON maintains the information in region abstraction that can save multiple pages with one data structure. The maximum number of regions can be set by users, so the space overhead can be controlled.
Quoted 2 lines
- Undoubtedly the mechanism to classify a page as hot and subsequent promotion needs to be more sophisticated than what I have right now.
DAMON provides aim-based DAMOS aggressiveness auto-tuning[6] and monitoring intervals auto-tuning[8] for this purpose.
Quoted 3 lines
This is just an early RFC posted now to ignite some discussion in the context of LSFMM [2].
This is really helpful. Appreciate, and looking forward to more discussions on LSFMM and mailing lists.
Quoted 3 lines
I am also working with Raghu to integrate his kmmdscan [3] as the hotness source and use kpromoted for migration.
Raghu also mentioned he would try to take a time to look into DAMON if there is anything that he could reuse for the purpose. I'm curious if he was able to find something there.
Quoted 4 lines
Also, I had posted the IBS driver ealier as an alternative to hint faults based NUMA Balancing [4]. However here I am using it as generic page hotness source.
This will also be very helpful for understanding how IBS can be used. Appreciate!
Quoted 5 lines
[1] https://lore.kernel.org/linux-mm/[email protected]/
[2] https://lore.kernel.org/linux-mm/[email protected]/
[3] https://lore.kernel.org/all/[email protected]/
[3] https://lore.kernel.org/lkml/[email protected]/
[1] https://github.com/skhynix/hmsdk/wiki/Capacity-Expansion
[2] https://lore.kernel.org/all/[email protected]/
[3] https://origin.kernel.org/doc/html/latest/mm/damon/design.html#operations-set-layer
[4] https://origin.kernel.org/doc/html/latest/mm/damon/design.html#operation-schemes
[5] https://origin.kernel.org/doc/html/latest/mm/damon/design.html#prioritization
[6] https://origin.kernel.org/doc/html/latest/mm/damon/design.html#aim-oriented-feedback-driven-auto-tuning
[7] https://lore.kernel.org/linux-mm/[email protected]/
[8] https://origin.kernel.org/doc/html/next/mm/damon/design.html#monitoring-intervals-auto-tuning
Thank,
SJ
Quoted 34 lines
Regards,
Bharata.Bharata B Rao (4):
mm: migrate: Allow misplaced migration without VMA too
mm: kpromoted: Hot page info collection and promotion daemon
x86: ibs: In-kernel IBS driver for memory access profiling
x86: ibs: Enable IBS profiling for memory accessesarch/x86/events/amd/ibs.c | 11 + arch/x86/include/asm/entry-common.h | 3 + arch/x86/include/asm/hardirq.h | 2 + arch/x86/include/asm/ibs.h | 9 + arch/x86/include/asm/msr-index.h | 16 ++ arch/x86/mm/Makefile | 3 +- arch/x86/mm/ibs.c | 344 ++++++++++++++++++++++++++++ include/linux/kpromoted.h | 54 +++++ include/linux/mmzone.h | 4 + include/linux/vm_event_item.h | 30 +++ mm/Kconfig | 7 + mm/Makefile | 1 + mm/kpromoted.c | 305 ++++++++++++++++++++++++ mm/migrate.c | 5 +- mm/mm_init.c | 10 + mm/vmstat.c | 30 +++ 16 files changed, 831 insertions(+), 3 deletions(-) create mode 100644 arch/x86/include/asm/ibs.h create mode 100644 arch/x86/mm/ibs.c create mode 100644 include/linux/kpromoted.h create mode 100644 mm/kpromoted.cSignature · 1 line
2.34.1
Quoted 9 lines · SeongJae Park
+ Harry, who was called Hyeonggon before.
Quoted 3 lines
I am also working with Raghu to integrate his kmmdscan [3] as the hotness source and use kpromoted for migration.
Raghu also mentioned he would try to take a time to look into DAMON if there is anything that he could reuse for the purpose. I'm curious if he was able to find something there.
[...]
Hello SJ,
I did take a look at DAMON vaddr and paddr implementation. Also wondering how can I optimize hotness data collected by kmmscand.
DAMON regions should be very helpful here, But I am not there yet.
will surely need help brainstorming session post my next RFC.
Thanks and Regards
- Raghu
Hi SJ,
Thanks for your detailed points and this surely sets up a good context for discussion in LSFMM.
Please see my replies to a few of your questions below:
Quoted 15 lines · SeongJae Park
Quoted 5 lines
Currently I have added AMD IBS driver as one source that provides page access information as an example. This driver feeds info to kpromoted in this RFC patchset. More sources were discussed in a similar context here at [1].
I was imagining how I would be able to do this with DAMON via operations set layer interface. And I find thee current interface is not very optimized for AMD IBS like sources that catches the access on the line. That is, in a way, we could say AMD IBS like primitives as push-oriented, while page tables' accessed bits information are pull-oriented. DAMON operations set layer interface is easier to be used in pull-oriented case. I don't think it cannot be used for push-oriented case, but definitely the interface would better to be more optimized for the use case.
Quoted 2 lines
I'm curious if you also tried doing this by extending DAMON, and if some hidden problems you found.
I remember discussing this with you during DAMON BoF in one of the earlier LPC events, but I didn't get to try it. Guess now is the time :-)
I see the challenge with the current DAMON interfaces to integrate IBS provided access info. If you check my IBS driver, I store the incoming access info from IBS into per-cpu buffers before pushing them on to the subsystem that act on them. I would think pull-based DAMON interfaces can consume those buffered samples rather than IBS pushing samples into DAMON. But I am yet to get clarity on how to honor the region based sampling that is inherent to DAMON's functioning. May be only using samples that are of interest to the region being tracked could be one way.
Quoted 33 lines
Quoted 7 lines
This is just an early attempt to check what it takes to maintain a single source of page hotness info and also separate hot page detection mechanisms from the promotion mechanism. There are too many open ends right now and I have listed a few of them below.
Quoted 10 lines
- The API that is provided to register memory access expects the PFN, NID and time of access at the minimum. This is described more in patch 2/4. This API currently can be called only from contexts that allow sleeping and hence this rules out using it from PTE scanning paths. The API needs to be more flexible with respect to this.
- Some sources like PTE A bit scanning can't provide the precise time of access or the NID that is accessing the page. The latter has been an open problem to which I haven't come across a good and acceptable solution.Agree. PTE A bit scanning could be useful in many cases, but not every case. There was an RFC patchset[7] that extends DAMON for NID. I'm planning to do that again using DAMON operations layer interface. My current plan is to implement the prototype using
prot_nonepage faults, and later extend for AMD IBS like h/w features. Hopefully I will share a prototype or at least more detailed idea on LSFMMBPF 2025.Quoted 4 lines
- The way the hot page information is maintained is pretty primitive right now. Ideally we would like to store hotness info in such a way that it should be easily possible to lookup say N most hot pages.
DAMON provides a feature for lookup of N most hotpages, namely DAMOS quotas' access pattern based regions prioritization[5].
Quoted 7 lines
Quoted 4 lines
- If PTE A bit scanners are considered as hotness sources, we will be bombarded with accesses. Do we want to accomodate all those accesses or just go with hotness info for fixed number of pages (possibly as a ratio of lower tier memory capacity)?
I understand you're saying about memory space overhead. Correct me if I'm wrong, please.
Correct and also the overhead of managing so much data. What I see is that if I start pushing all the access info obtained from LRU pgtable scanning, kpromoted would end up spending a lot of time in operations like lookup, walking the list of hot pages etc.
So may be it would be better to do some sort of early processing and/or filtering at the hotness source level itself before letting kpromoted-like subsystems to do further tracking and action.
Quoted 7 lines
Isn't same issue exists for current implementation of the sampling frequency is high, and/or aggregation window is long?
To me, hence, this looks like not a problem of the information source, but how to maintain the information. Current implementation maintains it per page, so I think the problem is inherent.
Well yes, but we the goal could be do better than NUMAB=2 which does per-page level tracking.
Quoted 4 lines
DAMON maintains the information in region abstraction that can save multiple pages with one data structure. The maximum number of regions can be set by users, so the space overhead can be controlled.
The granularity of tracking - per-page vs range/region is a topic of discussion I suppose.
Regards,
Bharata.
Quoted 15 lines · Bharata B Rao
Hi,
This is an attempt towards having a single subsystem that accumulates hot page information from lower memory tiers and does hot page promotion.
At the heart of this subsystem is a kernel daemon named kpromoted that does the following:
1. Exposes an API that other subsystems which detect/generate memory access information can use to inform the daemon about memory accesses from lower memory tiers.
2. Maintains the list of hot pages and attempts to promote them to toptiers.
Quoted 5 lines
Currently I have added AMD IBS driver as one source that provides page access information as an example. This driver feeds info to kpromoted in this RFC patchset. More sources were discussed in a similar context here at [1].
Is hot page promotion mandated or good to have? Memory tiers today are a function of latency and bandwidth, specifically in mt_aperf_to_distance()
adist ~ k * R(B)/R(L) where R(x) is relatively performance of the memory w.r.t DRAM. Do we want hot pages in the top tier all the time? Are we optimizing for bandwidth or latency?
Quoted 5 lines
This is just an early attempt to check what it takes to maintain a single source of page hotness info and also separate hot page detection mechanisms from the promotion mechanism. There are too many open ends right now and I have listed a few of them below.
<snip>
Quoted 3 lines
This is just an early RFC posted now to ignite some discussion in the context of LSFMM [2].
I look forward to any summary of the discussions
Balbir Singh
Hi Balbir,
Quoted 16 lines · Balbir Singh
Quoted 15 lines · Bharata B Rao
Hi,
This is an attempt towards having a single subsystem that accumulates hot page information from lower memory tiers and does hot page promotion.
At the heart of this subsystem is a kernel daemon named kpromoted that does the following:
1. Exposes an API that other subsystems which detect/generate memory access information can use to inform the daemon about memory accesses from lower memory tiers.
2. Maintains the list of hot pages and attempts to promote them to toptiers.
Quoted 7 lines
Quoted 5 lines
Currently I have added AMD IBS driver as one source that provides page access information as an example. This driver feeds info to kpromoted in this RFC patchset. More sources were discussed in a similar context here at [1].
Is hot page promotion mandated or good to have?
If you look at the current hot page promotion (NUMAB=2) logic, IIUC an accessed lower tier page is directly promoted to toptier if enough space exists in the toptier node. In such cases, it doesn't even bother about the hot threshold (measure of how recently it was accessed) or migration rate limiting. This tells me that it in a tiered memory setup, having an accessed page in toptier is preferrable.
Quoted 7 lines
Memory tiers today
are a function of latency and bandwidth, specifically inmt_aperf_to_distance()adist ~ k * R(B)/R(L) where R(x) is relatively performance of the memory w.r.t DRAM. Do we want hot pages in the top tier all the time? Are we optimizing for bandwidth or latency?
When memory tiering code converts BW and latency numbers into an opaque metric adistance based on which the node gets placed at an appropriate position in the tiering hierarchy, I wonder if it is still possible to say if we are optimizing for bandwidth or latency separately?
Quoted 7 lines
Quoted 5 lines
This is just an early attempt to check what it takes to maintain a single source of page hotness info and also separate hot page detection mechanisms from the promotion mechanism. There are too many open ends right now and I have listed a few of them below.
Quoted 7 lines
<snip>
Quoted 3 lines
This is just an early RFC posted now to ignite some discussion in the context of LSFMM [2].
I look forward to any summary of the discussions
Sure. Thanks,
Bharata.
Quoted 25 lines · Bharata B Rao
Hi Balbir,
Quoted 22 lines · Balbir Singh
Quoted 20 lines · Bharata B Rao
Hi,
This is an attempt towards having a single subsystem that accumulates hot page information from lower memory tiers and does hot page promotion.
At the heart of this subsystem is a kernel daemon named kpromoted that does the following:
1. Exposes an API that other subsystems which detect/generate memory access information can use to inform the daemon about memory accesses from lower memory tiers.
2. Maintains the list of hot pages and attempts to promote them to toptiers.Currently I have added AMD IBS driver as one source that provides page access information as an example. This driver feeds info to kpromoted in this RFC patchset. More sources were discussed in a similar context here at [1].
Quoted 4 lines
Quoted 1 line
Is hot page promotion mandated or good to have?
If you look at the current hot page promotion (NUMAB=2) logic, IIUC an accessed lower tier page is directly promoted to toptier if enough space exists in the toptier node. In such cases, it doesn't even bother about the hot threshold (measure of how recently it was accessed) or migration rate limiting. This tells me that it in a tiered memory setup, having an accessed page in toptier is preferrable.
I'll review the patches, I don't agree with toptier, I think DRAM is the right tier
Quoted 9 lines
Quoted 7 lines
Memory tiers today
are a function of latency and bandwidth, specifically inmt_aperf_to_distance()adist ~ k * R(B)/R(L) where R(x) is relatively performance of the memory w.r.t DRAM. Do we want hot pages in the top tier all the time? Are we optimizing for bandwidth or latency?
When memory tiering code converts BW and latency numbers into an opaque metric adistance based on which the node gets placed at an appropriate position in the tiering hierarchy, I wonder if it is still possible to say if we are optimizing for bandwidth or latency separately?
I think we need a notion of that, just higher tiers may not be right. IOW, I think we need to promote to at-most the DRAM tier, not above it.
Balbir Singh
Quoted 15 lines
Hi,
This is an attempt towards having a single subsystem that accumulates hot page information from lower memory tiers and does hot page promotion.
At the heart of this subsystem is a kernel daemon named kpromoted that does the following:
1. Exposes an API that other subsystems which detect/generate memory access information can use to inform the daemon about memory accesses from lower memory tiers.
2. Maintains the list of hot pages and attempts to promote them to toptiers.
Quoted 3 lines
Currently I have added AMD IBS driver as one source that provides page access information as an example. This driver feeds info to krpromoted in this RFC patchset.
FWIW, here are some numbers from krpomoted driven hotpage promotion with IBS as the hotness source:
Test 1
======
Memory allocated on DRAM and CXL nodes explicitly and no demotion activity is seen.
Benchmark details
-----------------
* Memory is allocated initially on DRAM and CXL nodes separately.
* Two threads: One accessing DRAM-allocated and other CXL-allocated memory.
* Divides memory area into regions and accesses pages within the region randomly and repetitively. In the test config shown below, the allocated memory is divided into regions of 1GB size and each such region is repetitively (512 times) accessed with 21474836480 random accesses in each repetition).
* Benchmark score is time taken for accesses to complete, lower is better
* Data accesses from CXL node are expected to trigger promotion
* Test system has 2 DRAM nodes (128G each) and a CXL node (128G)
kernel.numa_balancing 2 for base, 0 for kpromoted
demotion true
Threads run on Node 1
Memory allocated on Node 1(DRAM) and Node 2(CXL)
Initial allocation ratio 75% on DRAM
Allocated memory size 160G (mmap, MAP_POPULATE)
Initial memory on DRAM node 120G
Initial memory on CXL node 40G
Hot region size 1G
Acccess pattern random
Access granularity 4K
Load/store ratio 50% loads + 50% stores
Number of accesses 21474836480
Nr access repetitions 512
Benchmark completion time
-------------------------
Base, NUMAB=2 261s
kpromoted-ibs, NUMAB=0 281s
Stats comparision
-----------------
Base,NUMAB=2 kpromoted-IBS,NUMAB=0pgdemote_kswapd 0 0pgdemote_direct 0 0numa_pte_updates 10485760 0numa_hint_faults 4427809 0numa_pages_migrated 388229 374765kpromoted_recorded_accesses 1651130 /* nr accesses reported to kpromoted */kpromoted_recorded_hwhints 1651130 /* nr accesses coming from IBS */kpromoted_record_toptier 1269697 /* nr accesses from toptier/DRAM */kpromoted_record_added 378090 /* nr accesses considered for promotion */kpromoted_mig_promoted 374765 /* nr pages promoted */hwhint_nr_events 1674227 /* nr events reported by IBS */hwhint_dram_accesses 1269626 /* nr DRAM accesses reported by IBS */hwhint_cxl_accesses 381435 /* nr Extmem (CXL) accesses reported by IBS */hwhint_useful_samples 1651110 /* nr actionable samples as per IBS driver */
Test 2
======
Memory is allocated with DRAM and CXL nodes in the affinity mask with MPOL_BIND + MPOL_F_NUMA_BALANCING.
Benchmark details
-----------------
* Initially, memory allocated spreads over from DRAM to CXL, involves demotion
* Single thread accesses the memory
* Divides memory area into regions and accesses pages within the region randomly and repetitively. In the test config shown below, the allocated memory is divided into regions of 1GB size and each such region is repetitively (512 times) accessed with 21474836480 random accesses in each repetition).
* Benchmark score is time taken for accesses to complete, lower is better
* Data accesses from CXL node are expected to trigger promotion
* Test system has 2 DRAM nodes (128G each) and a CXL node (128G)
kernel.numa_balancing 2 for base, 0 for kpromoted
demotion true
Threads run on Node 1
Memory allocated on Node 1(DRAM) and Node 2(CXL)
Allocated memory size 192G (mmap, MAP_POPULATE)
Hot region size 1G
Acccess pattern random
Access granularity 4K
Load/store ratio 50% loads + 50% stores
Number of accesses 21474836480
Nr access repetitions 512
Benchmark completion time
-------------------------
Base, NUMAB=2 628s
kpromoted-ibs, NUMAB=0 626s
Stats comparision
-----------------
Base,NUMAB=2 kpromoted-IBS,NUMAB=0pgdemote_kswapd 73187 2196028pgdemote_direct 0 0numa_pte_updates 27511631 0numa_hint_faults 10010852 0numa_pages_migrated 14 611177 /* such low number of promotions is unexecpted in Base, Need to recheck */kpromoted_recorded_accesses 1883570kpromoted_recorded_hwhints 1883570kpromoted_record_toptier 1262088kpromoted_record_added 616273kpromoted_mig_promoted 611077hwhint_nr_events 1904619hwhint_dram_accesses 1261758hwhint_cxl_accesses 621428hwhint_useful_samples 1883543