Kernel thread based async batch migration
Subsystems (6)
cover Kernel thread based async batch migration
Hi,
This is a continuation of the earlier post[1] that attempted to convert migrations from NUMA Balancing to be async and batched. In this version, per-node kernel threads are created to handle migrations in an async manner.
This adds a few fields to the extended page flags that can be used both by the sub-systems that request migrations and kmigrated which migrates the pages. Some of the fields are potentially defined to be used by kpromoted-like subsystem to manage hot page metrics, but are unused right now.
Currently only NUMA Balancing is changed to make use of the async batched migration. It does so by recording the target NID and the readiness of the page to be migrated in the extended page flags fields.
Each kmigrated routinely scans its PFNs, identifies the pages marked for migration and batch-migrates them. Unlike the previous approach, the responsibility of isolating the pages is now with kmigrated.
The major difference between this approach and the way kpromoted[2] tracked hot pages is the elimination of heavy synchronization points between producers(sub-systems that request migrations or report a hot page) and the consumer (kmigrated or kpromoted). Instead of tracking only the list of hot pages in an orthogonal manner, this approach ties the hot page or migration infomation to the struct page.
TODOs:
- Very lightly tested(only with NUMAB=1) and posted to get some feedback on the overall approach.
- Currently uses the flags field from page extension sub-system. However need to check if it is preferrable to use/allocate a separate 32bit field exclusively for this purpose within page extension sub-system or outside of it.
- Benefit of async batch migration still needs to be measured.
- Need to really tune a few things like the number of pages to batch, the aggressiveness of kthread, the kthread sleep interval etc.
- The logic to skip scanning of zones that don't have any pages marked for migration needs to be added.
- No separate kernel config is defined currently and dependency on PAGE_EXTENSION isn't cleanly laid out. Some added definitions currently sit in page_ext.h which may not be an ideal location for them.
[1] v0 - https://lore.kernel.org/linux-mm/[email protected]/
[2] kpromoted patchset - https://lore.kernel.org/linux-mm/[email protected]/
Bharata B Rao (3):
mm: migrate: Allow misplaced migration without VMA too
mm: kmigrated - Async kernel migration thread
mm: sched: Batch-migrate misplaced pages
Gregory Price (1):
migrate: implement migrate_misplaced_folios_batch
include/linux/migrate.h | 6 ++ include/linux/mmzone.h | 5 + include/linux/page_ext.h | 17 +++ mm/Makefile | 3 +- mm/kmigrated.c | 223 +++++++++++++++++++++++++++++++++++++++ mm/memory.c | 30 +----- mm/migrate.c | 36 ++++++- mm/mm_init.c | 6 ++ mm/page_ext.c | 11 ++ 9 files changed, 309 insertions(+), 28 deletions(-) create mode 100644 mm/kmigrated.c
Signature · 1 line
2.34.1
RFC PATCH 1/4 mm: migrate: Allow misplaced migration without VMA too
We want isolation of misplaced folios to work in contexts where VMA isn't available. In order to prepare for that allow migrate_misplaced_folio_prepare() to be called with a NULL VMA.
--- mm/migrate.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
mm/migrate.c +3 -2
@@ -2580,7 +2580,8 @@ static struct folio *alloc_misplaced_dst_folio(struct folio *src,/* * Prepare for calling migrate_misplaced_folio() by isolating the folio if -* permitted. Must be called with the PTL still held. +* permitted. Must be called with the PTL still held if called with a non-NULL +* vma. */ int migrate_misplaced_folio_prepare (struct folio *folio,struct vm_area_struct *vma,int node)
@@ -2597,7 +2598,7 @@ int migrate_misplaced_folio_prepare(struct folio *folio,* See folio_maybe_mapped_shared() on possible imprecision * when we cannot easily detect if a folio is shared. */ -if ((vma->vm_flags &VM_EXEC ) &&folio_maybe_mapped_shared (folio)) +if (vma && (vma->vm_flags &VM_EXEC ) &&folio_maybe_mapped_shared (folio))return -EACCES ;/*
Signature · 1 line
2.34.1
RFC PATCH 2/4 migrate: implement migrate_misplaced_folios_batch
In reply to Bharata B RaoFrom: Gregory Price <[email protected]>
A common operation in tiering is to migrate multiple pages at once. The migrate_misplaced_folio function requires one call for each individual folio. Expose a batch-variant of the same call for use when doing batch migrations.
--- include/linux/migrate.h | 6 ++++++ mm/migrate.c | 31 +++++++++++++++++++++++++++++++ 2 files changed, 37 insertions(+)
include/linux/migrate.h +6 -0
@@ -145,6 +145,7 @@ const struct movable_operations *page_movable_ops(struct page *page)int migrate_misplaced_folio_prepare (struct folio *folio,struct vm_area_struct *vma,int node);int migrate_misplaced_folio (struct folio *folio,int node); +int migrate_misplaced_folios_batch (struct list_head *foliolist,int node);#else static inline int migrate_misplaced_folio_prepare (struct folio *folio,struct vm_area_struct *vma,int node)
@@ -155,6 +156,11 @@ static inline int migrate_misplaced_folio(struct folio *folio, int node) {return -EAGAIN ;/* can't migrate now */ } +static inline int migrate_misplaced_folios_batch (struct list_head *foliolist, +int node) +{ +return -EAGAIN ;/* can't migrate now */ +}#endif /* CONFIG_NUMA_BALANCING */ #ifdef CONFIG_MIGRATION
mm/migrate.c +31 -0
@@ -2675,5 +2675,36 @@ int migrate_misplaced_folio(struct folio *folio, int node)BUG_ON (!list_empty (&migratepages));return nr_remaining ? -EAGAIN :0 ; } + +/* +* Batch variant of migrate_misplaced_folio. Attempts to migrate +* a folio list to the specified destination. +* +* Caller is expected to have isolated the folios by calling +* migrate_misplaced_folio_prepare(), which will result in an +* elevated reference count on the folio. +* +* This function will un-isolate the folios, dereference them, and +* remove them from the list before returning. +*/ +int migrate_misplaced_folios_batch (struct list_head *folio_list,int node) +{ +pg_data_t *pgdat =NODE_DATA (node); +unsigned int nr_succeeded; +int nr_remaining; + + nr_remaining =migrate_pages (folio_list, alloc_misplaced_dst_folio, +NULL , node,MIGRATE_ASYNC , +MR_NUMA_MISPLACED , &nr_succeeded); +if (nr_remaining) +putback_movable_pages (folio_list); + +if (nr_succeeded) { +count_vm_numa_events (NUMA_PAGE_MIGRATE , nr_succeeded); +mod_node_page_state (pgdat,PGPROMOTE_SUCCESS , nr_succeeded); + } +BUG_ON (!list_empty (folio_list)); +return nr_remaining ? -EAGAIN :0 ; +}#endif /* CONFIG_NUMA_BALANCING */ #endif /* CONFIG_NUMA */
Signature · 1 line
2.34.1
RFC PATCH 3/4 mm: kmigrated - Async kernel migration thread
In reply to Bharata B Raokmigrated is a per-node kernel thread that migrates the folios marked for migration in batches. Each kmigrated thread walks the PFN range spanning its node and checks for potential migration candidates.
It depends on the fields added to extended page flags to determine the pages that need to be migrated and the target NID.
--- include/linux/mmzone.h | 5 + include/linux/page_ext.h | 17 +++ mm/Makefile | 3 +- mm/kmigrated.c | 223 +++++++++++++++++++++++++++++++++++++++ mm/mm_init.c | 6 ++ mm/page_ext.c | 11 ++ 6 files changed, 264 insertions(+), 1 deletion(-) create mode 100644 mm/kmigrated.c
include/linux/mmzone.h +5 -0
@@ -853,6 +853,8 @@ enum zone_type { }; +int kmigrated_add_pfn (unsigned long pfn,int nid); +#ifndef __GENERATING_BOUNDS_H#define ASYNC_AND_SYNC 2
@@ -1049,6 +1051,7 @@ enum pgdat_flags {* many pages under writeback */ PGDAT_RECLAIM_LOCKED ,/* prevents concurrent reclaim */ + PGDAT_KMIGRATED_ACTIVATE, /* activates kmigrated */ }; enum zone_flags {
@@ -1493,6 +1496,8 @@ typedef struct pglist_data {#ifdef CONFIG_MEMORY_FAILURE struct memory_failure_stats mf_stats;#endif +struct task_struct *kmigrated; +wait_queue_head_t kmigrated_wait; } pg_data_t;#define node_present_pages (nid) (NODE_DATA(nid)->node_present_pages)
include/linux/page_ext.h +17 -0
@@ -40,8 +40,25 @@ enum page_ext_flags {PAGE_EXT_YOUNG ,PAGE_EXT_IDLE , #endif +/* +* 32 bits following this are used by the migrator. +* The next available bit position is 33. +*/ + PAGE_EXT_MIGRATE_READY, }; +#define PAGE_EXT_MIG_NID_WIDTH 10 +#define PAGE_EXT_MIG_FREQ_WIDTH 3 +#define PAGE_EXT_MIG_TIME_WIDTH 18 + +#define PAGE_EXT_MIG_NID_SHIFT (PAGE_EXT_MIGRATE_READY + 1) +#define PAGE_EXT_MIG_FREQ_SHIFT (PAGE_EXT_MIG_NID_SHIFT + PAGE_EXT_MIG_NID_WIDTH) +#define PAGE_EXT_MIG_TIME_SHIFT (PAGE_EXT_MIG_FREQ_SHIFT + PAGE_EXT_MIG_FREQ_WIDTH) + +#define PAGE_EXT_MIG_NID_MASK ((1UL << PAGE_EXT_MIG_NID_SHIFT) - 1) +#define PAGE_EXT_MIG_FREQ_MASK ((1UL << PAGE_EXT_MIG_FREQ_SHIFT) - 1) +#define PAGE_EXT_MIG_TIME_MASK ((1UL << PAGE_EXT_MIG_TIME_SHIFT) - 1) +/* * Page Extension can be considered as an extended mem_map. * A page_ext page is associated with every page descriptor. The
mm/Makefile +2 -1
@@ -37,7 +37,8 @@ mmu-y := nommu.o mmu-$(CONFIG_MMU) := highmem.o memory.o mincore.o \ mlock.o mmap.o mmu_gather.o mprotect.o mremap.o \ msync.o page_vma_mapped.o pagewalk.o \ - pgtable-generic.o rmap.o vmalloc.o vma.o vma_exec.o + pgtable-generic.o rmap.o vmalloc.o vma.o vma_exec.o \ + kmigrated.o ifdef CONFIG_CROSS_MEMORY_ATTACH
mm/kmigrated.c new file +223 -0
223 lines not on this page · raw message
mm/mm_init.c +6 -0
@@ -1398,6 +1398,11 @@ static void pgdat_init_kcompactd(struct pglist_data *pgdat)static void pgdat_init_kcompactd (struct pglist_data *pgdat) {}#endif +static void pgdat_init_kmigrated (struct pglist_data *pgdat) +{ +init_waitqueue_head (&pgdat->kmigrated_wait); +} +static void __meminit pgdat_init_internals (struct pglist_data *pgdat) {int i;
@@ -1407,6 +1412,7 @@ static void __meminit pgdat_init_internals(struct pglist_data *pgdat)pgdat_init_split_queue (pgdat);pgdat_init_kcompactd (pgdat); +pgdat_init_kmigrated (pgdat);init_waitqueue_head (&pgdat->kswapd_wait);init_waitqueue_head (&pgdat->pfmemalloc_wait);
mm/page_ext.c +11 -0
@@ -76,6 +76,16 @@ static struct page_ext_operations page_idle_ops __initdata = { };#endif +static bool need_page_mig (void ) +{ +return true; +} + +static struct page_ext_operations page_mig_ops__initdata = { + .need = need_page_mig, + .need_shared_flags = true, +}; +static struct page_ext_operations *page_ext_ops[]__initdata = { #ifdef CONFIG_PAGE_OWNER &page_owner_ops,
@@ -89,6 +99,7 @@ static struct page_ext_operations *page_ext_ops[] __initdata = {#ifdef CONFIG_PAGE_TABLE_CHECK &page_table_check_ops, #endif + &page_mig_ops, };unsigned long page_ext_size;
Signature · 1 line
2.34.1
page_ext and memdescs
Quoted 16 lines · Bharata B Rao
include/linux/page_ext.h +6
-0@@ -40,8 +40,25 @@ enum page_ext_flags {PAGE_EXT_YOUNG ,PAGE_EXT_IDLE , #endif +/* +* 32 bits following this are used by the migrator. +* The next available bit position is 33. +*/ + PAGE_EXT_MIGRATE_READY, }; +#define PAGE_EXT_MIG_NID_WIDTH 10
include/linux/page_ext.h enum page_ext_flags { · line 60
Quoted 10 lines
include/linux/page_ext.h +10
-0+#define PAGE_EXT_MIG_FREQ_WIDTH 3 +#define PAGE_EXT_MIG_TIME_WIDTH 18 + +#define PAGE_EXT_MIG_NID_SHIFT (PAGE_EXT_MIGRATE_READY + 1) +#define PAGE_EXT_MIG_FREQ_SHIFT (PAGE_EXT_MIG_NID_SHIFT + PAGE_EXT_MIG_NID_WIDTH) +#define PAGE_EXT_MIG_TIME_SHIFT (PAGE_EXT_MIG_FREQ_SHIFT + PAGE_EXT_MIG_FREQ_WIDTH) + +#define PAGE_EXT_MIG_NID_MASK ((1UL << PAGE_EXT_MIG_NID_SHIFT) - 1) +#define PAGE_EXT_MIG_FREQ_MASK ((1UL << PAGE_EXT_MIG_FREQ_SHIFT) - 1) +#define PAGE_EXT_MIG_TIME_MASK ((1UL << PAGE_EXT_MIG_TIME_SHIFT) - 1)
OK, so we need to have a conversation about page_ext. Sorry this is happening to you. I've kind of skipped over page_ext when talking about folios and memdescs up to now, so it's not that you've missed anything.
As the comment says,
* Page Extension can be considered as an extended mem_map.
and we need to do this because we don't want to grow struct page beyond 64 bytes. But memdescs are dynamically allocated, so we don't need page_ext any more, and all that code can go away.
lib/alloc_tag.c:struct page_ext_operations page_alloc_tagging_ops = { mm/page_ext.c:static struct page_ext_operations page_idle_ops __initdata = { mm/page_ext.c:static struct page_ext_operations *page_ext_ops[] __initdata = { mm/page_owner.c:struct page_ext_operations page_owner_ops = { mm/page_table_check.c:struct page_ext_operations page_table_check_ops = {
I think all of these are actually per-memdesc thangs and not per-page things, so we can get rid of them all. That means I don't want to see new per-page data being added to page_ext.
So, what's this really used for? It seems like it's really per-allocation, not per-page. Does it need to be preserved across alloc/free or can it be reset at free time?
Re: page_ext and memdescs
Quoted 1 line · Matthew Wilcox
On Mon, Jun 16, 2025 at 07:09:30PM +0530, Bharata B Rao wrote:
<snip>
Quoted 23 lines
include/linux/page_ext.h enum page_ext_flags { · line 60
Quoted 3 lines
include/linux/page_ext.h +3
-0+#define PAGE_EXT_MIG_NID_MASK ((1UL << PAGE_EXT_MIG_NID_SHIFT) - 1) +#define PAGE_EXT_MIG_FREQ_MASK ((1UL << PAGE_EXT_MIG_FREQ_SHIFT) - 1) +#define PAGE_EXT_MIG_TIME_MASK ((1UL << PAGE_EXT_MIG_TIME_SHIFT) - 1)OK, so we need to have a conversation about
page_ext. Sorry this is happening to you. I've kind of skipped overpage_extwhen talking about folios and memdescs up to now, so it's not that you've missed anything.As the comment says,
* Page Extension can be considered as an extended
mem_map.and we need to do this because we don't want to grow struct page beyond 64 bytes. But memdescs are dynamically allocated, so we don't need
page_extany more, and all that code can go away.lib/alloc_tag.c:struct
page_ext_operationspage_alloc_tagging_ops= { mm/page_ext.c:static structpage_ext_operationspage_idle_ops__initdata= { mm/page_ext.c:static structpage_ext_operations*page_ext_ops[]__initdata= { mm/page_owner.c:structpage_ext_operationspage_owner_ops= { mm/page_table_check.c:structpage_ext_operationspage_table_check_ops= {
Quoted 3 lines
I think all of these are actually per-memdesc thangs and not per-page things, so we can get rid of them all. That means I don't want to see new per-page data being added to
page_ext.
Fair point.
Quoted 4 lines
So, what's this really used for? It seems like it's really per-allocation, not per-page. Does it need to be preserved across alloc/free or can it be reset at free time?
The context here is to track the pages that need to be migrated. Whether it is for NUMA Balancing or for any other sub-system that would need to migrate (or promote) pages across nodes, I am trying to come up with a kernel thread based migrator that would migrate the identified pages in an async and batched manner. For this, the basic information that is required for each such ready-to-be-migrated page is the target NID. Since I have chosen to walk the zones and PFNs of the zone to iterate over each page, an additional info that I want per ready-to-be-migrated page is an indication that the page is indeed ready now to be migrated by the migrator thread.
In addition to these two things, if we want to carve out a single system (like kpromoted approach) that handles inputs from multiple page hotness sources, maintains heuristics to decide when exactly to migrate/promote a page, then it would be good to store a few other information for such pages (like access frequency, access timestamp etc).
With that background, I am looking for an optimal place to store this information. In my earlier approaches, I was maintaining a global list of such hot pages and realized that such an approach will not scale and hence in the current approach I am tying that information with the page itself. With that, there is no overhead of maintaining such a list, synchronizing between producers and migrator thread, no allocation for each maintained page. Hence it appeared to me that a pre-allocated per-page info would be preferable. At this point, page extension appeared a good place to have this information.
Sorry for the long reply, but coming to your specific question now.
So I really need to maintain such data only for pages that can be migrated. Pages like most anonymous pages, file backed pages, pages that are mapped to user page tables, THP pages etc are candidates. I wonder which memdesc type/types would cover all such pages. Would "folio" as memdesc (https://kernelnewbies.org/MatthewWilcox/FolioAlloc) be broad enough type for this?
As you note, it appears to me that it could be per-allocation rather than per-page and the information needn't be preserved across alloc/free.
Regards,
Bharata.
Re: page_ext and memdescs
In reply to Matthew WilcoxQuoted 35 lines · Matthew Wilcox
Quoted 16 lines · Bharata B Rao
include/linux/page_ext.h +6
-0@@ -40,8 +40,25 @@ enum page_ext_flags {PAGE_EXT_YOUNG ,PAGE_EXT_IDLE , #endif +/* +* 32 bits following this are used by the migrator. +* The next available bit position is 33. +*/ + PAGE_EXT_MIGRATE_READY, }; +#define PAGE_EXT_MIG_NID_WIDTH 10include/linux/page_ext.h enum page_ext_flags { · line 60
Quoted 10 lines
include/linux/page_ext.h +10
-0+#define PAGE_EXT_MIG_FREQ_WIDTH 3 +#define PAGE_EXT_MIG_TIME_WIDTH 18 + +#define PAGE_EXT_MIG_NID_SHIFT (PAGE_EXT_MIGRATE_READY + 1) +#define PAGE_EXT_MIG_FREQ_SHIFT (PAGE_EXT_MIG_NID_SHIFT + PAGE_EXT_MIG_NID_WIDTH) +#define PAGE_EXT_MIG_TIME_SHIFT (PAGE_EXT_MIG_FREQ_SHIFT + PAGE_EXT_MIG_FREQ_WIDTH) + +#define PAGE_EXT_MIG_NID_MASK ((1UL << PAGE_EXT_MIG_NID_SHIFT) - 1) +#define PAGE_EXT_MIG_FREQ_MASK ((1UL << PAGE_EXT_MIG_FREQ_SHIFT) - 1) +#define PAGE_EXT_MIG_TIME_MASK ((1UL << PAGE_EXT_MIG_TIME_SHIFT) - 1)OK, so we need to have a conversation about
page_ext. Sorry this is happening to you. I've kind of skipped overpage_extwhen talking about folios and memdescs up to now, so it's not that you've missed anything.As the comment says,
Quoted 7 lines
* Page Extension can be considered as an extended
mem_map.and we need to do this because we don't want to grow struct page beyond 64 bytes. But memdescs are dynamically allocated, so we don't need
page_extany more, and all that code can go away.lib/alloc_tag.c:struct
page_ext_operationspage_alloc_tagging_ops= {
In this case, we might not necessarily have an allocated memdesc, for all allocations, though. Think of memory ballooning allocating "offline" pages in the future.
Of course, the easy solution is to not track these non-memdesc allocations.
Quoted 1 line
mm/page_ext.c:static struct
page_ext_operationspage_idle_ops__initdata= {
That should be per-folio.
Quoted 1 line
mm/page_ext.c:static struct
page_ext_operations*page_ext_ops[]__initdata= {
That's just the lookup table for the others.
Quoted 1 line
mm/page_owner.c:struct
page_ext_operationspage_owner_ops= {
Hm, probably like tagging above.
Quoted 1 line
mm/page_table_check.c:struct
page_ext_operationspage_table_check_ops= {
That should be per-folio as well IIUC.
Signature · 3 lines
Cheers, David / dhildenb
kernel test robot
build botRe: [RFC PATCH v1 3/4] mm: kmigrated - Async kernel migration thread
In reply to Bharata B Raokernel test robot noticed the following build errors:
Hi Bharata,
[This is a private test report for your RFC patch.] kernel test robot noticed the following build errors:
[auto build test ERROR on akpm-mm/mm-everything]
[also build test ERROR on linus/master v6.16-rc2 next-20250616] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Bharata-B-Rao/mm-migrate-Allow-misplaced-migration-without-VMA-too/20250616-214331 base: https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything patch link: https://lore.kernel.org/r/20250616133931.206626-4-bharata%40amd.com patch subject: [RFC PATCH v1 3/4] mm: kmigrated - Async kernel migration thread config: x86_64-buildonly-randconfig-005-20250617 (https://download.01.org/0day-ci/archive/20250617/[email protected]/config) compiler: gcc-12 (Debian 12.2.0-14) 12.2.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250617/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All error/warnings (new ones prefixed by >>):
In file included from <command-line>:
mm/kmigrated.c: In function 'page_ext_xchg_nid':
>> mm/kmigrated.c:26:39: error: invalid use of undefined type 'struct page_ext'
26 | old_flags = READ_ONCE(page_ext->flags);
| ^~
include/linux/compiler_types.h:548:23: note: in definition of macro '__compiletime_assert'
548 | if (!(condition)) \
| ^~~~~~~~~
include/linux/compiler_types.h:568:9: note: in expansion of macro '_compiletime_assert'
568 | _compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
| ^~~~~~~~~~~~~~~~~~~
include/asm-generic/rwonce.h:36:9: note: in expansion of macro 'compiletime_assert'
36 | compiletime_assert(__native_word(t) || sizeof(t) == sizeof(long long), \
| ^~~~~~~~~~~~~~~~~~
include/asm-generic/rwonce.h:36:28: note: in expansion of macro '__native_word'
36 | compiletime_assert(__native_word(t) || sizeof(t) == sizeof(long long), \
| ^~~~~~~~~~~~~
include/asm-generic/rwonce.h:49:9: note: in expansion of macro 'compiletime_assert_rwonce_type'
49 | compiletime_assert_rwonce_type(x); \
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
mm/kmigrated.c:26:21: note: in expansion of macro 'READ_ONCE'
26 | old_flags = READ_ONCE(page_ext->flags);
| ^~~~~~~~~
>> mm/kmigrated.c:26:39: error: invalid use of undefined type 'struct page_ext'
26 | old_flags = READ_ONCE(page_ext->flags);
| ^~
include/linux/compiler_types.h:548:23: note: in definition of macro '__compiletime_assert'
548 | if (!(condition)) \
| ^~~~~~~~~
include/linux/compiler_types.h:568:9: note: in expansion of macro '_compiletime_assert'
568 | _compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
| ^~~~~~~~~~~~~~~~~~~
include/asm-generic/rwonce.h:36:9: note: in expansion of macro 'compiletime_assert'
36 | compiletime_assert(__native_word(t) || sizeof(t) == sizeof(long long), \
| ^~~~~~~~~~~~~~~~~~
include/asm-generic/rwonce.h:36:28: note: in expansion of macro '__native_word'
36 | compiletime_assert(__native_word(t) || sizeof(t) == sizeof(long long), \
| ^~~~~~~~~~~~~
include/asm-generic/rwonce.h:49:9: note: in expansion of macro 'compiletime_assert_rwonce_type'
49 | compiletime_assert_rwonce_type(x); \
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
mm/kmigrated.c:26:21: note: in expansion of macro 'READ_ONCE'
26 | old_flags = READ_ONCE(page_ext->flags);
| ^~~~~~~~~
>> mm/kmigrated.c:26:39: error: invalid use of undefined type 'struct page_ext'
26 | old_flags = READ_ONCE(page_ext->flags);
| ^~
include/linux/compiler_types.h:548:23: note: in definition of macro '__compiletime_assert'
548 | if (!(condition)) \
| ^~~~~~~~~
include/linux/compiler_types.h:568:9: note: in expansion of macro '_compiletime_assert'
568 | _compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
| ^~~~~~~~~~~~~~~~~~~
include/asm-generic/rwonce.h:36:9: note: in expansion of macro 'compiletime_assert'
36 | compiletime_assert(__native_word(t) || sizeof(t) == sizeof(long long), \
| ^~~~~~~~~~~~~~~~~~
include/asm-generic/rwonce.h:36:28: note: in expansion of macro '__native_word'
36 | compiletime_assert(__native_word(t) || sizeof(t) == sizeof(long long), \
| ^~~~~~~~~~~~~
include/asm-generic/rwonce.h:49:9: note: in expansion of macro 'compiletime_assert_rwonce_type'
49 | compiletime_assert_rwonce_type(x); \
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
mm/kmigrated.c:26:21: note: in expansion of macro 'READ_ONCE'
26 | old_flags = READ_ONCE(page_ext->flags);
| ^~~~~~~~~
>> mm/kmigrated.c:26:39: error: invalid use of undefined type 'struct page_ext'
26 | old_flags = READ_ONCE(page_ext->flags);
| ^~
include/linux/compiler_types.h:548:23: note: in definition of macro '__compiletime_assert'
548 | if (!(condition)) \
| ^~~~~~~~~
include/linux/compiler_types.h:568:9: note: in expansion of macro '_compiletime_assert'
568 | _compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
| ^~~~~~~~~~~~~~~~~~~
include/asm-generic/rwonce.h:36:9: note: in expansion of macro 'compiletime_assert'
36 | compiletime_assert(__native_word(t) || sizeof(t) == sizeof(long long), \
| ^~~~~~~~~~~~~~~~~~
include/asm-generic/rwonce.h:36:28: note: in expansion of macro '__native_word'
36 | compiletime_assert(__native_word(t) || sizeof(t) == sizeof(long long), \
| ^~~~~~~~~~~~~
include/asm-generic/rwonce.h:49:9: note: in expansion of macro 'compiletime_assert_rwonce_type'
49 | compiletime_assert_rwonce_type(x); \
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
mm/kmigrated.c:26:21: note: in expansion of macro 'READ_ONCE'
26 | old_flags = READ_ONCE(page_ext->flags);
| ^~~~~~~~~
>> mm/kmigrated.c:26:39: error: invalid use of undefined type 'struct page_ext'
26 | old_flags = READ_ONCE(page_ext->flags);
| ^~
include/linux/compiler_types.h:548:23: note: in definition of macro '__compiletime_assert'
548 | if (!(condition)) \
| ^~~~~~~~~
include/linux/compiler_types.h:568:9: note: in expansion of macro '_compiletime_assert'
568 | _compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
| ^~~~~~~~~~~~~~~~~~~
include/asm-generic/rwonce.h:36:9: note: in expansion of macro 'compiletime_assert'
36 | compiletime_assert(__native_word(t) || sizeof(t) == sizeof(long long), \
| ^~~~~~~~~~~~~~~~~~
include/asm-generic/rwonce.h:49:9: note: in expansion of macro 'compiletime_assert_rwonce_type'
49 | compiletime_assert_rwonce_type(x); \
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
mm/kmigrated.c:26:21: note: in expansion of macro 'READ_ONCE'
26 | old_flags = READ_ONCE(page_ext->flags);
| ^~~~~~~~~
>> mm/kmigrated.c:26:39: error: invalid use of undefined type 'struct page_ext'
26 | old_flags = READ_ONCE(page_ext->flags);
| ^~
include/linux/compiler_types.h:518:27: note: in definition of macro '__unqual_scalar_typeof'
518 | _Generic((x), \
| ^
include/asm-generic/rwonce.h:50:9: note: in expansion of macro '__READ_ONCE'
50 | __READ_ONCE(x); \
| ^~~~~~~~~~~
mm/kmigrated.c:26:21: note: in expansion of macro 'READ_ONCE'
26 | old_flags = READ_ONCE(page_ext->flags);
| ^~~~~~~~~
In file included from ./arch/x86/include/generated/asm/rwonce.h:1,
from include/linux/compiler.h:390,
from include/asm-generic/bug.h:5,
from arch/x86/include/asm/bug.h:103,
from include/linux/bug.h:5,
from include/linux/mmdebug.h:5,
from include/linux/mm.h:6,
from mm/kmigrated.c:13:
>> mm/kmigrated.c:26:39: error: invalid use of undefined type 'struct page_ext'
26 | old_flags = READ_ONCE(page_ext->flags);
| ^~
include/asm-generic/rwonce.h:44:73: note: in definition of macro '__READ_ONCE'
44 | #define __READ_ONCE(x) (*(const volatile __unqual_scalar_typeof(x) *)&(x))
| ^
mm/kmigrated.c:26:21: note: in expansion of macro 'READ_ONCE'
26 | old_flags = READ_ONCE(page_ext->flags);
| ^~~~~~~~~
>> mm/kmigrated.c:29:37: error: 'PAGE_EXT_MIG_NID_SHIFT' undeclared (first use in this function)
29 | old_nid = (flags >> PAGE_EXT_MIG_NID_SHIFT) & PAGE_EXT_MIG_NID_MASK;
| ^~~~~~~~~~~~~~~~~~~~~~
mm/kmigrated.c:29:37: note: each undeclared identifier is reported only once for each function it appears in
>> mm/kmigrated.c:29:63: error: 'PAGE_EXT_MIG_NID_MASK' undeclared (first use in this function)
29 | old_nid = (flags >> PAGE_EXT_MIG_NID_SHIFT) & PAGE_EXT_MIG_NID_MASK;
| ^~~~~~~~~~~~~~~~~~~~~
mm/kmigrated.c:33:49: error: invalid use of undefined type 'struct page_ext'
33 | } while (unlikely(!try_cmpxchg(&page_ext->flags, &old_flags, flags)));
| ^~
include/linux/compiler.h:32:55: note: in definition of macro '__branch_check__'
32 | ______r = __builtin_expect(!!(x), expect); \
| ^
mm/kmigrated.c:33:18: note: in expansion of macro 'unlikely'
33 | } while (unlikely(!try_cmpxchg(&page_ext->flags, &old_flags, flags)));
| ^~~~~~~~
mm/kmigrated.c:33:28: note: in expansion of macro 'try_cmpxchg'
33 | } while (unlikely(!try_cmpxchg(&page_ext->flags, &old_flags, flags)));
| ^~~~~~~~~~~
mm/kmigrated.c:33:49: error: invalid use of undefined type 'struct page_ext'
33 | } while (unlikely(!try_cmpxchg(&page_ext->flags, &old_flags, flags)));
| ^~
include/linux/compiler.h:32:55: note: in definition of macro '__branch_check__'
32 | ______r = __builtin_expect(!!(x), expect); \
| ^
mm/kmigrated.c:33:18: note: in expansion of macro 'unlikely'
33 | } while (unlikely(!try_cmpxchg(&page_ext->flags, &old_flags, flags)));
| ^~~~~~~~
mm/kmigrated.c:33:28: note: in expansion of macro 'try_cmpxchg'
33 | } while (unlikely(!try_cmpxchg(&page_ext->flags, &old_flags, flags)));
| ^~~~~~~~~~~
include/linux/atomic/atomic-instrumented.h:4878:55: error: invalid type argument of unary '*' (have 'int')
4878 | instrument_atomic_read_write(__ai_ptr, sizeof(*__ai_ptr)); \
| ^~~~~~~~~
include/linux/compiler.h:32:55: note: in definition of macro '__branch_check__'
32 | ______r = __builtin_expect(!!(x), expect); \
| ^
mm/kmigrated.c:33:18: note: in expansion of macro 'unlikely'
33 | } while (unlikely(!try_cmpxchg(&page_ext->flags, &old_flags, flags)));
| ^~~~~~~~
mm/kmigrated.c:33:28: note: in expansion of macro 'try_cmpxchg'
33 | } while (unlikely(!try_cmpxchg(&page_ext->flags, &old_flags, flags)));
| ^~~~~~~~~~~
>> include/linux/atomic/atomic-instrumented.h:4878:38: warning: passing argument 1 of 'instrument_atomic_read_write' makes pointer from integer without a cast [-Wint-conversion]
4878 | instrument_atomic_read_write(__ai_ptr, sizeof(*__ai_ptr)); \
| ^~~~~~~~
| |
| int
include/linux/compiler.h:32:55: note: in definition of macro '__branch_check__'
32 | ______r = __builtin_expect(!!(x), expect); \
| ^
mm/kmigrated.c:33:18: note: in expansion of macro 'unlikely'
33 | } while (unlikely(!try_cmpxchg(&page_ext->flags, &old_flags, flags)));
| ^~~~~~~~
mm/kmigrated.c:33:28: note: in expansion of macro 'try_cmpxchg'
33 | } while (unlikely(!try_cmpxchg(&page_ext->flags, &old_flags, flags)));
| ^~~~~~~~~~~
In file included from include/asm-generic/bitops/instrumented-atomic.h:14,
from arch/x86/include/asm/bitops.h:429,
from include/linux/bitops.h:67,
from include/linux/thread_info.h:27,
from include/linux/spinlock.h:60,
from include/linux/mmzone.h:8,
from include/linux/gfp.h:7,
from include/linux/mm.h:7:
include/linux/instrumented.h:94:79: note: expected 'const volatile void *' but argument is of type 'int'
94 | static __always_inline void instrument_atomic_read_write(const volatile void *v, size_t size)
| ~~~~~~~~~~~~~~~~~~~~~^
arch/x86/include/asm/cmpxchg.h:161:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast]
161 | __typeof__(_ptr) _old = (__typeof__(_ptr))(_pold); \
| ^
include/linux/compiler.h:32:55: note: in definition of macro '__branch_check__'
32 | ______r = __builtin_expect(!!(x), expect); \
| ^
mm/kmigrated.c:33:18: note: in expansion of macro 'unlikely'
33 | } while (unlikely(!try_cmpxchg(&page_ext->flags, &old_flags, flags)));
| ^~~~~~~~
arch/x86/include/asm/cmpxchg.h:222:9: note: in expansion of macro '__raw_try_cmpxchg'
222 | __raw_try_cmpxchg((ptr), (pold), (new), (size), LOCK_PREFIX)
| ^~~~~~~~~~~~~~~~~
arch/x86/include/asm/cmpxchg.h:231:9: note: in expansion of macro '__try_cmpxchg'
231 | __try_cmpxchg((ptr), (pold), (new), sizeof(*(ptr)))
| ^~~~~~~~~~~~~
include/linux/atomic/atomic-arch-fallback.h:184:25: note: in expansion of macro 'arch_try_cmpxchg'
184 | #define raw_try_cmpxchg arch_try_cmpxchg
| ^~~~~~~~~~~~~~~~
include/linux/atomic/atomic-instrumented.h:4880:9: note: in expansion of macro 'raw_try_cmpxchg'
4880 | raw_try_cmpxchg(__ai_ptr, __ai_oldp, __VA_ARGS__); \
| ^~~~~~~~~~~~~~~
mm/kmigrated.c:33:28: note: in expansion of macro 'try_cmpxchg'
33 | } while (unlikely(!try_cmpxchg(&page_ext->flags, &old_flags, flags)));
| ^~~~~~~~~~~
arch/x86/include/asm/cmpxchg.h:162:20: error: invalid type argument of unary '*' (have 'int')
162 | __typeof__(*(_ptr)) __old = *_old; \
| ^~~~~~~
include/linux/compiler.h:32:55: note: in definition of macro '__branch_check__'
32 | ______r = __builtin_expect(!!(x), expect); \
| ^
mm/kmigrated.c:33:18: note: in expansion of macro 'unlikely'
33 | } while (unlikely(!try_cmpxchg(&page_ext->flags, &old_flags, flags)));
| ^~~~~~~~
arch/x86/include/asm/cmpxchg.h:222:9: note: in expansion of macro '__raw_try_cmpxchg'
222 | __raw_try_cmpxchg((ptr), (pold), (new), (size), LOCK_PREFIX)
| ^~~~~~~~~~~~~~~~~
arch/x86/include/asm/cmpxchg.h:231:9: note: in expansion of macro '__try_cmpxchg'
231 | __try_cmpxchg((ptr), (pold), (new), sizeof(*(ptr)))
| ^~~~~~~~~~~~~
include/linux/atomic/atomic-arch-fallback.h:184:25: note: in expansion of macro 'arch_try_cmpxchg'
184 | #define raw_try_cmpxchg arch_try_cmpxchg
| ^~~~~~~~~~~~~~~~
include/linux/atomic/atomic-instrumented.h:4880:9: note: in expansion of macro 'raw_try_cmpxchg'
4880 | raw_try_cmpxchg(__ai_ptr, __ai_oldp, __VA_ARGS__); \
| ^~~~~~~~~~~~~~~
mm/kmigrated.c:33:28: note: in expansion of macro 'try_cmpxchg'
33 | } while (unlikely(!try_cmpxchg(&page_ext->flags, &old_flags, flags)));
| ^~~~~~~~~~~
arch/x86/include/asm/cmpxchg.h:162:37: error: invalid type argument of unary '*' (have 'int')
162 | __typeof__(*(_ptr)) __old = *_old; \
| ^~~~~
include/linux/compiler.h:32:55: note: in definition of macro '__branch_check__'
32 | ______r = __builtin_expect(!!(x), expect); \
| ^
mm/kmigrated.c:33:18: note: in expansion of macro 'unlikely'
33 | } while (unlikely(!try_cmpxchg(&page_ext->flags, &old_flags, flags)));
| ^~~~~~~~
arch/x86/include/asm/cmpxchg.h:222:9: note: in expansion of macro '__raw_try_cmpxchg'
222 | __raw_try_cmpxchg((ptr), (pold), (new), (size), LOCK_PREFIX)
| ^~~~~~~~~~~~~~~~~
arch/x86/include/asm/cmpxchg.h:231:9: note: in expansion of macro '__try_cmpxchg'
231 | __try_cmpxchg((ptr), (pold), (new), sizeof(*(ptr)))
| ^~~~~~~~~~~~~
include/linux/atomic/atomic-arch-fallback.h:184:25: note: in expansion of macro 'arch_try_cmpxchg'
184 | #define raw_try_cmpxchg arch_try_cmpxchg
| ^~~~~~~~~~~~~~~~
include/linux/atomic/atomic-instrumented.h:4880:9: note: in expansion of macro 'raw_try_cmpxchg'
4880 | raw_try_cmpxchg(__ai_ptr, __ai_oldp, __VA_ARGS__); \
| ^~~~~~~~~~~~~~~
mm/kmigrated.c:33:28: note: in expansion of macro 'try_cmpxchg'
33 | } while (unlikely(!try_cmpxchg(&page_ext->flags, &old_flags, flags)));
| ^~~~~~~~~~~
arch/x86/include/asm/cmpxchg.h:163:20: error: invalid type argument of unary '*' (have 'int')
163 | __typeof__(*(_ptr)) __new = (_new); \
| ^~~~~~~
include/linux/compiler.h:32:55: note: in definition of macro '__branch_check__'
32 | ______r = __builtin_expect(!!(x), expect); \
| ^
mm/kmigrated.c:33:18: note: in expansion of macro 'unlikely'
33 | } while (unlikely(!try_cmpxchg(&page_ext->flags, &old_flags, flags)));
| ^~~~~~~~
arch/x86/include/asm/cmpxchg.h:222:9: note: in expansion of macro '__raw_try_cmpxchg'
222 | __raw_try_cmpxchg((ptr), (pold), (new), (size), LOCK_PREFIX)
| ^~~~~~~~~~~~~~~~~
arch/x86/include/asm/cmpxchg.h:231:9: note: in expansion of macro '__try_cmpxchg'
231 | __try_cmpxchg((ptr), (pold), (new), sizeof(*(ptr)))
| ^~~~~~~~~~~~~
include/linux/atomic/atomic-arch-fallback.h:184:25: note: in expansion of macro 'arch_try_cmpxchg'
184 | #define raw_try_cmpxchg arch_try_cmpxchg
| ^~~~~~~~~~~~~~~~
include/linux/atomic/atomic-instrumented.h:4880:9: note: in expansion of macro 'raw_try_cmpxchg'
4880 | raw_try_cmpxchg(__ai_ptr, __ai_oldp, __VA_ARGS__); \
| ^~~~~~~~~~~~~~~
mm/kmigrated.c:33:28: note: in expansion of macro 'try_cmpxchg'
33 | } while (unlikely(!try_cmpxchg(&page_ext->flags, &old_flags, flags)));
| ^~~~~~~~~~~
arch/x86/include/asm/cmpxchg.h:231:52: error: invalid type argument of unary '*' (have 'int')
231 | __try_cmpxchg((ptr), (pold), (new), sizeof(*(ptr)))
| ^~~~~~
include/linux/compiler.h:32:55: note: in definition of macro '__branch_check__'
32 | ______r = __builtin_expect(!!(x), expect); \
| ^
mm/kmigrated.c:33:18: note: in expansion of macro 'unlikely'
33 | } while (unlikely(!try_cmpxchg(&page_ext->flags, &old_flags, flags)));
| ^~~~~~~~
arch/x86/include/asm/cmpxchg.h:222:9: note: in expansion of macro '__raw_try_cmpxchg'
222 | __raw_try_cmpxchg((ptr), (pold), (new), (size), LOCK_PREFIX)
| ^~~~~~~~~~~~~~~~~
arch/x86/include/asm/cmpxchg.h:231:9: note: in expansion of macro '__try_cmpxchg'
231 | __try_cmpxchg((ptr), (pold), (new), sizeof(*(ptr)))
| ^~~~~~~~~~~~~
include/linux/atomic/atomic-arch-fallback.h:184:25: note: in expansion of macro 'arch_try_cmpxchg'
184 | #define raw_try_cmpxchg arch_try_cmpxchg
| ^~~~~~~~~~~~~~~~
include/linux/atomic/atomic-instrumented.h:4880:9: note: in expansion of macro 'raw_try_cmpxchg'
4880 | raw_try_cmpxchg(__ai_ptr, __ai_oldp, __VA_ARGS__); \
| ^~~~~~~~~~~~~~~
mm/kmigrated.c:33:28: note: in expansion of macro 'try_cmpxchg'
33 | } while (unlikely(!try_cmpxchg(&page_ext->flags, &old_flags, flags)));
| ^~~~~~~~~~~
>> arch/x86/include/asm/cmpxchg.h:167:38: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
167 | volatile u8 *__ptr = (volatile u8 *)(_ptr); \
| ^
include/linux/compiler.h:32:55: note: in definition of macro '__branch_check__'
32 | ______r = __builtin_expect(!!(x), expect); \
| ^
mm/kmigrated.c:33:18: note: in expansion of macro 'unlikely'
33 | } while (unlikely(!try_cmpxchg(&page_ext->flags, &old_flags, flags)));
| ^~~~~~~~
arch/x86/include/asm/cmpxchg.h:222:9: note: in expansion of macro '__raw_try_cmpxchg'
222 | __raw_try_cmpxchg((ptr), (pold), (new), (size), LOCK_PREFIX)
| ^~~~~~~~~~~~~~~~~
arch/x86/include/asm/cmpxchg.h:231:9: note: in expansion of macro '__try_cmpxchg'
231 | __try_cmpxchg((ptr), (pold), (new), sizeof(*(ptr)))
| ^~~~~~~~~~~~~
include/linux/atomic/atomic-arch-fallback.h:184:25: note: in expansion of macro 'arch_try_cmpxchg'
184 | #define raw_try_cmpxchg arch_try_cmpxchg
| ^~~~~~~~~~~~~~~~
include/linux/atomic/atomic-instrumented.h:4880:9: note: in expansion of macro 'raw_try_cmpxchg'
4880 | raw_try_cmpxchg(__ai_ptr, __ai_oldp, __VA_ARGS__); \
| ^~~~~~~~~~~~~~~
mm/kmigrated.c:33:28: note: in expansion of macro 'try_cmpxchg'
33 | } while (unlikely(!try_cmpxchg(&page_ext->flags, &old_flags, flags)));
| ^~~~~~~~~~~
arch/x86/include/asm/cmpxchg.h:179:39: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
179 | volatile u16 *__ptr = (volatile u16 *)(_ptr); \
| ^
include/linux/compiler.h:32:55: note: in definition of macro '__branch_check__'
32 | ______r = __builtin_expect(!!(x), expect); \
| ^
mm/kmigrated.c:33:18: note: in expansion of macro 'unlikely'
33 | } while (unlikely(!try_cmpxchg(&page_ext->flags, &old_flags, flags)));
| ^~~~~~~~
arch/x86/include/asm/cmpxchg.h:222:9: note: in expansion of macro '__raw_try_cmpxchg'
222 | __raw_try_cmpxchg((ptr), (pold), (new), (size), LOCK_PREFIX)
| ^~~~~~~~~~~~~~~~~
arch/x86/include/asm/cmpxchg.h:231:9: note: in expansion of macro '__try_cmpxchg'
231 | __try_cmpxchg((ptr), (pold), (new), sizeof(*(ptr)))
| ^~~~~~~~~~~~~
include/linux/atomic/atomic-arch-fallback.h:184:25: note: in expansion of macro 'arch_try_cmpxchg'
184 | #define raw_try_cmpxchg arch_try_cmpxchg
| ^~~~~~~~~~~~~~~~
include/linux/atomic/atomic-instrumented.h:4880:9: note: in expansion of macro 'raw_try_cmpxchg'
4880 | raw_try_cmpxchg(__ai_ptr, __ai_oldp, __VA_ARGS__); \
| ^~~~~~~~~~~~~~~
mm/kmigrated.c:33:28: note: in expansion of macro 'try_cmpxchg'
33 | } while (unlikely(!try_cmpxchg(&page_ext->flags, &old_flags, flags)));
| ^~~~~~~~~~~
arch/x86/include/asm/cmpxchg.h:191:39: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
191 | volatile u32 *__ptr = (volatile u32 *)(_ptr); \
| ^
include/linux/compiler.h:32:55: note: in definition of macro '__branch_check__'
32 | ______r = __builtin_expect(!!(x), expect); \
| ^
mm/kmigrated.c:33:18: note: in expansion of macro 'unlikely'
33 | } while (unlikely(!try_cmpxchg(&page_ext->flags, &old_flags, flags)));
| ^~~~~~~~
arch/x86/include/asm/cmpxchg.h:222:9: note: in expansion of macro '__raw_try_cmpxchg'
222 | __raw_try_cmpxchg((ptr), (pold), (new), (size), LOCK_PREFIX)
| ^~~~~~~~~~~~~~~~~
arch/x86/include/asm/cmpxchg.h:231:9: note: in expansion of macro '__try_cmpxchg'
231 | __try_cmpxchg((ptr), (pold), (new), sizeof(*(ptr)))
| ^~~~~~~~~~~~~
include/linux/atomic/atomic-arch-fallback.h:184:25: note: in expansion of macro 'arch_try_cmpxchg'
184 | #define raw_try_cmpxchg arch_try_cmpxchg
| ^~~~~~~~~~~~~~~~
include/linux/atomic/atomic-instrumented.h:4880:9: note: in expansion of macro 'raw_try_cmpxchg'
4880 | raw_try_cmpxchg(__ai_ptr, __ai_oldp, __VA_ARGS__); \
| ^~~~~~~~~~~~~~~
mm/kmigrated.c:33:28: note: in expansion of macro 'try_cmpxchg'
33 | } while (unlikely(!try_cmpxchg(&page_ext->flags, &old_flags, flags)));
| ^~~~~~~~~~~
arch/x86/include/asm/cmpxchg.h:203:39: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
203 | volatile u64 *__ptr = (volatile u64 *)(_ptr); \
| ^
include/linux/compiler.h:32:55: note: in definition of macro '__branch_check__'
32 | ______r = __builtin_expect(!!(x), expect); \
| ^
mm/kmigrated.c:33:18: note: in expansion of macro 'unlikely'
33 | } while (unlikely(!try_cmpxchg(&page_ext->flags, &old_flags, flags)));
| ^~~~~~~~
arch/x86/include/asm/cmpxchg.h:222:9: note: in expansion of macro '__raw_try_cmpxchg'
222 | __raw_try_cmpxchg((ptr), (pold), (new), (size), LOCK_PREFIX)
| ^~~~~~~~~~~~~~~~~
arch/x86/include/asm/cmpxchg.h:231:9: note: in expansion of macro '__try_cmpxchg'
231 | __try_cmpxchg((ptr), (pold), (new), sizeof(*(ptr)))
| ^~~~~~~~~~~~~
include/linux/atomic/atomic-arch-fallback.h:184:25: note: in expansion of macro 'arch_try_cmpxchg'
184 | #define raw_try_cmpxchg arch_try_cmpxchg
| ^~~~~~~~~~~~~~~~
include/linux/atomic/atomic-instrumented.h:4880:9: note: in expansion of macro 'raw_try_cmpxchg'
4880 | raw_try_cmpxchg(__ai_ptr, __ai_oldp, __VA_ARGS__); \
| ^~~~~~~~~~~~~~~
mm/kmigrated.c:33:28: note: in expansion of macro 'try_cmpxchg'
33 | } while (unlikely(!try_cmpxchg(&page_ext->flags, &old_flags, flags)));
| ^~~~~~~~~~~
arch/x86/include/asm/cmpxchg.h:217:17: error: invalid type argument of unary '*' (have 'int')
217 | *_old = __old; \
| ^~~~~
include/linux/compiler.h:32:55: note: in definition of macro '__branch_check__'
32 | ______r = __builtin_expect(!!(x), expect); \
| ^
mm/kmigrated.c:33:18: note: in expansion of macro 'unlikely'
33 | } while (unlikely(!try_cmpxchg(&page_ext->flags, &old_flags, flags)));
| ^~~~~~~~
arch/x86/include/asm/cmpxchg.h:222:9: note: in expansion of macro '__raw_try_cmpxchg'
222 | __raw_try_cmpxchg((ptr), (pold), (new), (size), LOCK_PREFIX)
| ^~~~~~~~~~~~~~~~~
arch/x86/include/asm/cmpxchg.h:231:9: note: in expansion of macro '__try_cmpxchg'
231 | __try_cmpxchg((ptr), (pold), (new), sizeof(*(ptr)))
| ^~~~~~~~~~~~~
include/linux/atomic/atomic-arch-fallback.h:184:25: note: in expansion of macro 'arch_try_cmpxchg'
184 | #define raw_try_cmpxchg arch_try_cmpxchg
| ^~~~~~~~~~~~~~~~
include/linux/atomic/atomic-instrumented.h:4880:9: note: in expansion of macro 'raw_try_cmpxchg'
4880 | raw_try_cmpxchg(__ai_ptr, __ai_oldp, __VA_ARGS__); \
| ^~~~~~~~~~~~~~~
mm/kmigrated.c:33:28: note: in expansion of macro 'try_cmpxchg'
33 | } while (unlikely(!try_cmpxchg(&page_ext->flags, &old_flags, flags)));
| ^~~~~~~~~~~
mm/kmigrated.c:33:49: error: invalid use of undefined type 'struct page_ext'
33 | } while (unlikely(!try_cmpxchg(&page_ext->flags, &old_flags, flags)));
| ^~
include/linux/compiler.h:34:54: note: in definition of macro '__branch_check__'
34 | expect, is_constant); \
| ^~~~~~~~~~~
mm/kmigrated.c:33:18: note: in expansion of macro 'unlikely'
33 | } while (unlikely(!try_cmpxchg(&page_ext->flags, &old_flags, flags)));
| ^~~~~~~~
mm/kmigrated.c:33:28: note: in expansion of macro 'try_cmpxchg'
33 | } while (unlikely(!try_cmpxchg(&page_ext->flags, &old_flags, flags)));
| ^~~~~~~~~~~
mm/kmigrated.c:33:49: error: invalid use of undefined type 'struct page_ext'
33 | } while (unlikely(!try_cmpxchg(&page_ext->flags, &old_flags, flags)));
| ^~
include/linux/compiler.h:34:54: note: in definition of macro '__branch_check__'
34 | expect, is_constant); \
| ^~~~~~~~~~~
mm/kmigrated.c:33:18: note: in expansion of macro 'unlikely'
33 | } while (unlikely(!try_cmpxchg(&page_ext->flags, &old_flags, flags)));
| ^~~~~~~~
mm/kmigrated.c:33:28: note: in expansion of macro 'try_cmpxchg'
33 | } while (unlikely(!try_cmpxchg(&page_ext->flags, &old_flags, flags)));
| ^~~~~~~~~~~
include/linux/atomic/atomic-instrumented.h:4878:55: error: invalid type argument of unary '*' (have 'int')
4878 | instrument_atomic_read_write(__ai_ptr, sizeof(*__ai_ptr)); \
| ^~~~~~~~~
include/linux/compiler.h:34:54: note: in definition of macro '__branch_check__'
34 | expect, is_constant); \
| ^~~~~~~~~~~
mm/kmigrated.c:33:18: note: in expansion of macro 'unlikely'
33 | } while (unlikely(!try_cmpxchg(&page_ext->flags, &old_flags, flags)));
| ^~~~~~~~
mm/kmigrated.c:33:28: note: in expansion of macro 'try_cmpxchg'
33 | } while (unlikely(!try_cmpxchg(&page_ext->flags, &old_flags, flags)));
| ^~~~~~~~~~~
>> include/linux/atomic/atomic-instrumented.h:4878:38: warning: passing argument 1 of 'instrument_atomic_read_write' makes pointer from integer without a cast [-Wint-conversion]
4878 | instrument_atomic_read_write(__ai_ptr, sizeof(*__ai_ptr)); \
| ^~~~~~~~
| |
| int
include/linux/compiler.h:34:54: note: in definition of macro '__branch_check__'
34 | expect, is_constant); \
| ^~~~~~~~~~~
mm/kmigrated.c:33:18: note: in expansion of macro 'unlikely'
33 | } while (unlikely(!try_cmpxchg(&page_ext->flags, &old_flags, flags)));
| ^~~~~~~~
mm/kmigrated.c:33:28: note: in expansion of macro 'try_cmpxchg'
33 | } while (unlikely(!try_cmpxchg(&page_ext->flags, &old_flags, flags)));
| ^~~~~~~~~~~
include/linux/instrumented.h:94:79: note: expected 'const volatile void *' but argument is of type 'int'
94 | static __always_inline void instrument_atomic_read_write(const volatile void *v, size_t size)
| ~~~~~~~~~~~~~~~~~~~~~^
arch/x86/include/asm/cmpxchg.h:161:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast]
161 | __typeof__(_ptr) _old = (__typeof__(_ptr))(_pold); \
| ^
include/linux/compiler.h:34:54: note: in definition of macro '__branch_check__'
34 | expect, is_constant); \
| ^~~~~~~~~~~
mm/kmigrated.c:33:18: note: in expansion of macro 'unlikely'
33 | } while (unlikely(!try_cmpxchg(&page_ext->flags, &old_flags, flags)));
| ^~~~~~~~
arch/x86/include/asm/cmpxchg.h:222:9: note: in expansion of macro '__raw_try_cmpxchg'
222 | __raw_try_cmpxchg((ptr), (pold), (new), (size), LOCK_PREFIX)
| ^~~~~~~~~~~~~~~~~
arch/x86/include/asm/cmpxchg.h:231:9: note: in expansion of macro '__try_cmpxchg'
231 | __try_cmpxchg((ptr), (pold), (new), sizeof(*(ptr)))
| ^~~~~~~~~~~~~
include/linux/atomic/atomic-arch-fallback.h:184:25: note: in expansion of macro 'arch_try_cmpxchg'
184 | #define raw_try_cmpxchg arch_try_cmpxchg
| ^~~~~~~~~~~~~~~~
include/linux/atomic/atomic-instrumented.h:4880:9: note: in expansion of macro 'raw_try_cmpxchg'
4880 | raw_try_cmpxchg(__ai_ptr, __ai_oldp, __VA_ARGS__); \
| ^~~~~~~~~~~~~~~
mm/kmigrated.c:33:28: note: in expansion of macro 'try_cmpxchg'
33 | } while (unlikely(!try_cmpxchg(&page_ext->flags, &old_flags, flags)));
| ^~~~~~~~~~~
arch/x86/include/asm/cmpxchg.h:162:20: error: invalid type argument of unary '*' (have 'int')
162 | __typeof__(*(_ptr)) __old = *_old; \
| ^~~~~~~
include/linux/compiler.h:34:54: note: in definition of macro '__branch_check__'
34 | expect, is_constant); \
| ^~~~~~~~~~~
mm/kmigrated.c:33:18: note: in expansion of macro 'unlikely'
33 | } while (unlikely(!try_cmpxchg(&page_ext->flags, &old_flags, flags)));
| ^~~~~~~~
arch/x86/include/asm/cmpxchg.h:222:9: note: in expansion of macro '__raw_try_cmpxchg'
222 | __raw_try_cmpxchg((ptr), (pold), (new), (size), LOCK_PREFIX)
| ^~~~~~~~~~~~~~~~~
arch/x86/include/asm/cmpxchg.h:231:9: note: in expansion of macro '__try_cmpxchg'
231 | __try_cmpxchg((ptr), (pold), (new), sizeof(*(ptr)))
| ^~~~~~~~~~~~~
include/linux/atomic/atomic-arch-fallback.h:184:25: note: in expansion of macro 'arch_try_cmpxchg'
184 | #define raw_try_cmpxchg arch_try_cmpxchg
| ^~~~~~~~~~~~~~~~
include/linux/atomic/atomic-instrumented.h:4880:9: note: in expansion of macro 'raw_try_cmpxchg'
4880 | raw_try_cmpxchg(__ai_ptr, __ai_oldp, __VA_ARGS__); \
| ^~~~~~~~~~~~~~~
mm/kmigrated.c:33:28: note: in expansion of macro 'try_cmpxchg'
33 | } while (unlikely(!try_cmpxchg(&page_ext->flags, &old_flags, flags)));
| ^~~~~~~~~~~
arch/x86/include/asm/cmpxchg.h:162:37: error: invalid type argument of unary '*' (have 'int')
162 | __typeof__(*(_ptr)) __old = *_old; \
| ^~~~~
include/linux/compiler.h:34:54: note: in definition of macro '__branch_check__'
34 | expect, is_constant); \
| ^~~~~~~~~~~
mm/kmigrated.c:33:18: note: in expansion of macro 'unlikely'
33 | } while (unlikely(!try_cmpxchg(&page_ext->flags, &old_flags, flags)));
| ^~~~~~~~
arch/x86/include/asm/cmpxchg.h:222:9: note: in expansion of macro '__raw_try_cmpxchg'
222 | __raw_try_cmpxchg((ptr), (pold), (new), (size), LOCK_PREFIX)
| ^~~~~~~~~~~~~~~~~
arch/x86/include/asm/cmpxchg.h:231:9: note: in expansion of macro '__try_cmpxchg'
231 | __try_cmpxchg((ptr), (pold), (new), sizeof(*(ptr)))
| ^~~~~~~~~~~~~
include/linux/atomic/atomic-arch-fallback.h:184:25: note: in expansion of macro 'arch_try_cmpxchg'
184 | #define raw_try_cmpxchg arch_try_cmpxchg
| ^~~~~~~~~~~~~~~~
include/linux/atomic/atomic-instrumented.h:4880:9: note: in expansion of macro 'raw_try_cmpxchg'
4880 | raw_try_cmpxchg(__ai_ptr, __ai_oldp, __VA_ARGS__); \
| ^~~~~~~~~~~~~~~
mm/kmigrated.c:33:28: note: in expansion of macro 'try_cmpxchg'
33 | } while (unlikely(!try_cmpxchg(&page_ext->flags, &old_flags, flags)));
| ^~~~~~~~~~~
arch/x86/include/asm/cmpxchg.h:163:20: error: invalid type argument of unary '*' (have 'int')
163 | __typeof__(*(_ptr)) __new = (_new); \
| ^~~~~~~
include/linux/compiler.h:34:54: note: in definition of macro '__branch_check__'
34 | expect, is_constant); \
| ^~~~~~~~~~~
mm/kmigrated.c:33:18: note: in expansion of macro 'unlikely'
33 | } while (unlikely(!try_cmpxchg(&page_ext->flags, &old_flags, flags)));
| ^~~~~~~~
arch/x86/include/asm/cmpxchg.h:222:9: note: in expansion of macro '__raw_try_cmpxchg'
222 | __raw_try_cmpxchg((ptr), (pold), (new), (size), LOCK_PREFIX)
| ^~~~~~~~~~~~~~~~~
arch/x86/include/asm/cmpxchg.h:231:9: note: in expansion of macro '__try_cmpxchg'
231 | __try_cmpxchg((ptr), (pold), (new), sizeof(*(ptr)))
| ^~~~~~~~~~~~~
include/linux/atomic/atomic-arch-fallback.h:184:25: note: in expansion of macro 'arch_try_cmpxchg'
184 | #define raw_try_cmpxchg arch_try_cmpxchg
| ^~~~~~~~~~~~~~~~
include/linux/atomic/atomic-instrumented.h:4880:9: note: in expansion of macro 'raw_try_cmpxchg'
4880 | raw_try_cmpxchg(__ai_ptr, __ai_oldp, __VA_ARGS__); \
| ^~~~~~~~~~~~~~~
mm/kmigrated.c:33:28: note: in expansion of macro 'try_cmpxchg'
33 | } while (unlikely(!try_cmpxchg(&page_ext->flags, &old_flags, flags)));
| ^~~~~~~~~~~
arch/x86/include/asm/cmpxchg.h:231:52: error: invalid type argument of unary '*' (have 'int')
231 | __try_cmpxchg((ptr), (pold), (new), sizeof(*(ptr)))
| ^~~~~~
include/linux/compiler.h:34:54: note: in definition of macro '__branch_check__'
34 | expect, is_constant); \
| ^~~~~~~~~~~
mm/kmigrated.c:33:18: note: in expansion of macro 'unlikely'
33 | } while (unlikely(!try_cmpxchg(&page_ext->flags, &old_flags, flags)));
| ^~~~~~~~
arch/x86/include/asm/cmpxchg.h:222:9: note: in expansion of macro '__raw_try_cmpxchg'
222 | __raw_try_cmpxchg((ptr), (pold), (new), (size), LOCK_PREFIX)
| ^~~~~~~~~~~~~~~~~
arch/x86/include/asm/cmpxchg.h:231:9: note: in expansion of macro '__try_cmpxchg'
231 | __try_cmpxchg((ptr), (pold), (new), sizeof(*(ptr)))
| ^~~~~~~~~~~~~
include/linux/atomic/atomic-arch-fallback.h:184:25: note: in expansion of macro 'arch_try_cmpxchg'
184 | #define raw_try_cmpxchg arch_try_cmpxchg
| ^~~~~~~~~~~~~~~~
include/linux/atomic/atomic-instrumented.h:4880:9: note: in expansion of macro 'raw_try_cmpxchg'
4880 | raw_try_cmpxchg(__ai_ptr, __ai_oldp, __VA_ARGS__); \
| ^~~~~~~~~~~~~~~
mm/kmigrated.c:33:28: note: in expansion of macro 'try_cmpxchg'
33 | } while (unlikely(!try_cmpxchg(&page_ext->flags, &old_flags, flags)));
| ^~~~~~~~~~~
>> arch/x86/include/asm/cmpxchg.h:167:38: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
167 | volatile u8 *__ptr = (volatile u8 *)(_ptr); \
| ^
include/linux/compiler.h:34:54: note: in definition of macro '__branch_check__'
34 | expect, is_constant); \
| ^~~~~~~~~~~
mm/kmigrated.c:33:18: note: in expansion of macro 'unlikely'
33 | } while (unlikely(!try_cmpxchg(&page_ext->flags, &old_flags, flags)));
| ^~~~~~~~
arch/x86/include/asm/cmpxchg.h:222:9: note: in expansion of macro '__raw_try_cmpxchg'
222 | __raw_try_cmpxchg((ptr), (pold), (new), (size), LOCK_PREFIX)
| ^~~~~~~~~~~~~~~~~
arch/x86/include/asm/cmpxchg.h:231:9: note: in expansion of macro '__try_cmpxchg'
231 | __try_cmpxchg((ptr), (pold), (new), sizeof(*(ptr)))
| ^~~~~~~~~~~~~
include/linux/atomic/atomic-arch-fallback.h:184:25: note: in expansion of macro 'arch_try_cmpxchg'
184 | #define raw_try_cmpxchg arch_try_cmpxchg
| ^~~~~~~~~~~~~~~~
include/linux/atomic/atomic-instrumented.h:4880:9: note: in expansion of macro 'raw_try_cmpxchg'
4880 | raw_try_cmpxchg(__ai_ptr, __ai_oldp, __VA_ARGS__); \
| ^~~~~~~~~~~~~~~
mm/kmigrated.c:33:28: note: in expansion of macro 'try_cmpxchg'
33 | } while (unlikely(!try_cmpxchg(&page_ext->flags, &old_flags, flags)));
| ^~~~~~~~~~~
arch/x86/include/asm/cmpxchg.h:179:39: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
179 | volatile u16 *__ptr = (volatile u16 *)(_ptr); \
| ^
include/linux/compiler.h:34:54: note: in definition of macro '__branch_check__'
34 | expect, is_constant); \
| ^~~~~~~~~~~
mm/kmigrated.c:33:18: note: in expansion of macro 'unlikely'
33 | } while (unlikely(!try_cmpxchg(&page_ext->flags, &old_flags, flags)));
| ^~~~~~~~
arch/x86/include/asm/cmpxchg.h:222:9: note: in expansion of macro '__raw_try_cmpxchg'
222 | __raw_try_cmpxchg((ptr), (pold), (new), (size), LOCK_PREFIX)
| ^~~~~~~~~~~~~~~~~
arch/x86/include/asm/cmpxchg.h:231:9: note: in expansion of macro '__try_cmpxchg'
231 | __try_cmpxchg((ptr), (pold), (new), sizeof(*(ptr)))
| ^~~~~~~~~~~~~
include/linux/atomic/atomic-arch-fallback.h:184:25: note: in expansion of macro 'arch_try_cmpxchg'
184 | #define raw_try_cmpxchg arch_try_cmpxchg
| ^~~~~~~~~~~~~~~~
include/linux/atomic/atomic-instrumented.h:4880:9: note: in expansion of macro 'raw_try_cmpxchg'
4880 | raw_try_cmpxchg(__ai_ptr, __ai_oldp, __VA_ARGS__); \
| ^~~~~~~~~~~~~~~
mm/kmigrated.c:33:28: note: in expansion of macro 'try_cmpxchg'
33 | } while (unlikely(!try_cmpxchg(&page_ext->flags, &old_flags, flags)));
| ^~~~~~~~~~~
arch/x86/include/asm/cmpxchg.h:191:39: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
191 | volatile u32 *__ptr = (volatile u32 *)(_ptr); \
| ^
include/linux/compiler.h:34:54: note: in definition of macro '__branch_check__'
34 | expect, is_constant); \
| ^~~~~~~~~~~
mm/kmigrated.c:33:18: note: in expansion of macro 'unlikely'
33 | } while (unlikely(!try_cmpxchg(&page_ext->flags, &old_flags, flags)));
| ^~~~~~~~
arch/x86/include/asm/cmpxchg.h:222:9: note: in expansion of macro '__raw_try_cmpxchg'
222 | __raw_try_cmpxchg((ptr), (pold), (new), (size), LOCK_PREFIX)
| ^~~~~~~~~~~~~~~~~
arch/x86/include/asm/cmpxchg.h:231:9: note: in expansion of macro '__try_cmpxchg'
231 | __try_cmpxchg((ptr), (pold), (new), sizeof(*(ptr)))
| ^~~~~~~~~~~~~
include/linux/atomic/atomic-arch-fallback.h:184:25: note: in expansion of macro 'arch_try_cmpxchg'
184 | #define raw_try_cmpxchg arch_try_cmpxchg
| ^~~~~~~~~~~~~~~~
include/linux/atomic/atomic-instrumented.h:4880:9: note: in expansion of macro 'raw_try_cmpxchg'
4880 | raw_try_cmpxchg(__ai_ptr, __ai_oldp, __VA_ARGS__); \
| ^~~~~~~~~~~~~~~
mm/kmigrated.c:33:28: note: in expansion of macro 'try_cmpxchg'
33 | } while (unlikely(!try_cmpxchg(&page_ext->flags, &old_flags, flags)));
| ^~~~~~~~~~~
arch/x86/include/asm/cmpxchg.h:203:39: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
203 | volatile u64 *__ptr = (volatile u64 *)(_ptr); \
| ^
include/linux/compiler.h:34:54: note: in definition of macro '__branch_check__'
34 | expect, is_constant); \
| ^~~~~~~~~~~
mm/kmigrated.c:33:18: note: in expansion of macro 'unlikely'
33 | } while (unlikely(!try_cmpxchg(&page_ext->flags, &old_flags, flags)));
| ^~~~~~~~
arch/x86/include/asm/cmpxchg.h:222:9: note: in expansion of macro '__raw_try_cmpxchg'
222 | __raw_try_cmpxchg((ptr), (pold), (new), (size), LOCK_PREFIX)
| ^~~~~~~~~~~~~~~~~
arch/x86/include/asm/cmpxchg.h:231:9: note: in expansion of macro '__try_cmpxchg'
231 | __try_cmpxchg((ptr), (pold), (new), sizeof(*(ptr)))
| ^~~~~~~~~~~~~
include/linux/atomic/atomic-arch-fallback.h:184:25: note: in expansion of macro 'arch_try_cmpxchg'
184 | #define raw_try_cmpxchg arch_try_cmpxchg
| ^~~~~~~~~~~~~~~~
include/linux/atomic/atomic-instrumented.h:4880:9: note: in expansion of macro 'raw_try_cmpxchg'
4880 | raw_try_cmpxchg(__ai_ptr, __ai_oldp, __VA_ARGS__); \
| ^~~~~~~~~~~~~~~
mm/kmigrated.c:33:28: note: in expansion of macro 'try_cmpxchg'
33 | } while (unlikely(!try_cmpxchg(&page_ext->flags, &old_flags, flags)));
| ^~~~~~~~~~~
arch/x86/include/asm/cmpxchg.h:217:17: error: invalid type argument of unary '*' (have 'int')
217 | *_old = __old; \
| ^~~~~
include/linux/compiler.h:34:54: note: in definition of macro '__branch_check__'
34 | expect, is_constant); \
| ^~~~~~~~~~~
mm/kmigrated.c:33:18: note: in expansion of macro 'unlikely'
33 | } while (unlikely(!try_cmpxchg(&page_ext->flags, &old_flags, flags)));
| ^~~~~~~~
arch/x86/include/asm/cmpxchg.h:222:9: note: in expansion of macro '__raw_try_cmpxchg'
222 | __raw_try_cmpxchg((ptr), (pold), (new), (size), LOCK_PREFIX)
| ^~~~~~~~~~~~~~~~~
arch/x86/include/asm/cmpxchg.h:231:9: note: in expansion of macro '__try_cmpxchg'
231 | __try_cmpxchg((ptr), (pold), (new), sizeof(*(ptr)))
| ^~~~~~~~~~~~~
include/linux/atomic/atomic-arch-fallback.h:184:25: note: in expansion of macro 'arch_try_cmpxchg'
184 | #define raw_try_cmpxchg arch_try_cmpxchg
| ^~~~~~~~~~~~~~~~
include/linux/atomic/atomic-instrumented.h:4880:9: note: in expansion of macro 'raw_try_cmpxchg'
4880 | raw_try_cmpxchg(__ai_ptr, __ai_oldp, __VA_ARGS__); \
| ^~~~~~~~~~~~~~~
mm/kmigrated.c:33:28: note: in expansion of macro 'try_cmpxchg'
33 | } while (unlikely(!try_cmpxchg(&page_ext->flags, &old_flags, flags)));
| ^~~~~~~~~~~
mm/kmigrated.c: In function 'kmigrated_add_pfn':
>> mm/kmigrated.c:61:26: error: 'PAGE_EXT_MIGRATE_READY' undeclared (first use in this function)
61 | test_and_set_bit(PAGE_EXT_MIGRATE_READY, &page_ext->flags);
| ^~~~~~~~~~~~~~~~~~~~~~
mm/kmigrated.c:61:59: error: invalid use of undefined type 'struct page_ext'
61 | test_and_set_bit(PAGE_EXT_MIGRATE_READY, &page_ext->flags);
| ^~
mm/kmigrated.c: In function 'kmigrated_get_nid':
mm/kmigrated.c:83:33: error: 'PAGE_EXT_MIGRATE_READY' undeclared (first use in this function)
83 | if (!test_and_clear_bit(PAGE_EXT_MIGRATE_READY, &page_ext->flags))
| ^~~~~~~~~~~~~~~~~~~~~~
mm/kmigrated.c:83:66: error: invalid use of undefined type 'struct page_ext'
83 | if (!test_and_clear_bit(PAGE_EXT_MIGRATE_READY, &page_ext->flags))
| ^~vim +26 mm/kmigrated.c
20
21 static int page_ext_xchg_nid(struct page_ext *page_ext, int nid)
22 {
23 unsigned long old_flags, flags;
24 int old_nid;
25
> 26 old_flags = READ_ONCE(page_ext->flags);
27 do {
28 flags = old_flags;
> 29 old_nid = (flags >> PAGE_EXT_MIG_NID_SHIFT) & PAGE_EXT_MIG_NID_MASK;
30
31 flags &= ~(PAGE_EXT_MIG_NID_MASK << PAGE_EXT_MIG_NID_SHIFT);
32 flags |= (nid & PAGE_EXT_MIG_NID_MASK) << PAGE_EXT_MIG_NID_SHIFT;
33 } while (unlikely(!try_cmpxchg(&page_ext->flags, &old_flags, flags)));
34
35 return old_nid;
36 }
37
38 /*
39 * Marks the page as ready for migration.
40 *
41 * @pfn: PFN of the page
42 * @nid: Target NID to were the page needs to be migrated
43 *
44 * The request for migration is noted by setting PAGE_EXT_MIGRATE_READY
45 * in the extended page flags which the kmigrated thread would check.
46 */
47 int kmigrated_add_pfn(unsigned long pfn, int nid)
48 {
49 struct page *page;
50 struct page_ext *page_ext;
51
52 page = pfn_to_page(pfn);
53 if (!page)
54 return -EINVAL;
55
56 page_ext = page_ext_get(page);
57 if (unlikely(!page_ext))
58 return -EINVAL;
59
60 page_ext_xchg_nid(page_ext, nid);
> 61 test_and_set_bit(PAGE_EXT_MIGRATE_READY, &page_ext->flags);
62 page_ext_put(page_ext);
63
64 set_bit(PGDAT_KMIGRATED_ACTIVATE, &page_pgdat(page)->flags);
65 return 0;
66 }
67 Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Byungchul Park
reviewerRe: [RFC PATCH v1 3/4] mm: kmigrated - Async kernel migration thread
In reply to Bharata B RaoQuoted 196 lines · Bharata B Rao
kmigrated is a per-node kernel thread that migrates the folios marked for migration in batches. Each kmigrated thread walks the PFN range spanning its node and checks for potential migration candidates.
It depends on the fields added to extended page flags to determine the pages that need to be migrated and the target NID.
Signed-off-by Bharata B Rao <[email protected]>--- include/linux/mmzone.h | 5 + include/linux/page_ext.h | 17 +++ mm/Makefile | 3 +- mm/kmigrated.c | 223 +++++++++++++++++++++++++++++++++++++++ mm/mm_init.c | 6 ++ mm/page_ext.c | 11 ++ 6 files changed, 264 insertions(+), 1 deletion(-) create mode 100644 mm/kmigrated.cinclude/linux/mmzone.h +5
-0@@ -853,6 +853,8 @@ enum zone_type { }; +int kmigrated_add_pfn (unsigned long pfn,int nid); +#ifndef __GENERATING_BOUNDS_H#define ASYNC_AND_SYNC 2@@ -1049,6 +1051,7 @@ enum pgdat_flags {* many pages under writeback */ PGDAT_RECLAIM_LOCKED ,/* prevents concurrent reclaim */ + PGDAT_KMIGRATED_ACTIVATE, /* activates kmigrated */ }; enum zone_flags {@@ -1493,6 +1496,8 @@ typedef struct pglist_data {#ifdef CONFIG_MEMORY_FAILURE struct memory_failure_stats mf_stats;#endif +struct task_struct *kmigrated; +wait_queue_head_t kmigrated_wait; } pg_data_t;#define node_present_pages (nid) (NODE_DATA(nid)->node_present_pages)include/linux/page_ext.h +17
-0@@ -40,8 +40,25 @@ enum page_ext_flags {PAGE_EXT_YOUNG ,PAGE_EXT_IDLE , #endif +/* +* 32 bits following this are used by the migrator. +* The next available bit position is 33. +*/ + PAGE_EXT_MIGRATE_READY, }; +#define PAGE_EXT_MIG_NID_WIDTH 10 +#define PAGE_EXT_MIG_FREQ_WIDTH 3 +#define PAGE_EXT_MIG_TIME_WIDTH 18 + +#define PAGE_EXT_MIG_NID_SHIFT (PAGE_EXT_MIGRATE_READY + 1) +#define PAGE_EXT_MIG_FREQ_SHIFT (PAGE_EXT_MIG_NID_SHIFT + PAGE_EXT_MIG_NID_WIDTH) +#define PAGE_EXT_MIG_TIME_SHIFT (PAGE_EXT_MIG_FREQ_SHIFT + PAGE_EXT_MIG_FREQ_WIDTH) + +#define PAGE_EXT_MIG_NID_MASK ((1UL << PAGE_EXT_MIG_NID_SHIFT) - 1) +#define PAGE_EXT_MIG_FREQ_MASK ((1UL << PAGE_EXT_MIG_FREQ_SHIFT) - 1) +#define PAGE_EXT_MIG_TIME_MASK ((1UL << PAGE_EXT_MIG_TIME_SHIFT) - 1) +/* * Page Extension can be considered as an extended mem_map. * A page_ext page is associated with every page descriptor. The mm/Makefile +2
-1@@ -37,7 +37,8 @@ mmu-y := nommu.o mmu-$(CONFIG_MMU) := highmem.o memory.o mincore.o \ mlock.o mmap.o mmu_gather.o mprotect.o mremap.o \ msync.o page_vma_mapped.o pagewalk.o \ - pgtable-generic.o rmap.o vmalloc.o vma.o vma_exec.o + pgtable-generic.o rmap.o vmalloc.o vma.o vma_exec.o \ + kmigrated.o ifdef CONFIG_CROSS_MEMORY_ATTACHmm/kmigrated.c new file +95
-0@@ -0,0 +1,223 @@ +// SPDX-License-Identifier: GPL-2.0 +/* +* kmigrated is a kernel thread that runs for each node that has +* memory. It iterates over the node's PFNs and migrates pages +* marked for migration into their targeted nodes. +* +* kmigrated depends on PAGE_EXTENSION to find out the pages that +* need to be migrated. In addition to a few fields that could be +* used by hot page promotion logic to store and evaluate the page +* hotness information, the extended page flags is field is extended +* to store the target NID for migration. +*/ +#include <linux/mm.h> +#include <linux/migrate.h> +#include <linux/cpuhotplug.h> +#include <linux/page_ext.h> + +#define KMIGRATE_DELAY MSEC_PER_SEC +#define KMIGRATE_BATCH 512 + +static int page_ext_xchg_nid (struct page_ext *page_ext,int nid) +{ +unsigned long old_flags, flags; +int old_nid; + + old_flags =READ_ONCE (page_ext->flags); +do { + flags = old_flags; + old_nid = (flags >>PAGE_EXT_MIG_NID_SHIFT ) &PAGE_EXT_MIG_NID_MASK ; + + flags &= ~(PAGE_EXT_MIG_NID_MASK <<PAGE_EXT_MIG_NID_SHIFT ); + flags |= (nid &PAGE_EXT_MIG_NID_MASK ) <<PAGE_EXT_MIG_NID_SHIFT ; + }while (unlikely (!try_cmpxchg (&page_ext->flags, &old_flags, flags))); + +return old_nid; +} + +/* +* Marks the page as ready for migration. +* +* @pfn: PFN of the page +* @nid: Target NID to were the page needs to be migrated +* +* The request for migration is noted by setting PAGE_EXT_MIGRATE_READY +* in the extended page flags which the kmigrated thread would check. +*/ +int kmigrated_add_pfn (unsigned long pfn,int nid) +{ +struct page *page; +struct page_ext *page_ext; + + page =pfn_to_page (pfn); +if (!page) +return -EINVAL ; + + page_ext =page_ext_get (page); +if (unlikely (!page_ext)) +return -EINVAL ; + +page_ext_xchg_nid (page_ext, nid); +test_and_set_bit (PAGE_EXT_MIGRATE_READY , &page_ext->flags); +page_ext_put (page_ext); + +set_bit (PGDAT_KMIGRATED_ACTIVATE , &page_pgdat (page)->flags); +return 0 ; +} + +/* +* If the page has been marked ready for migration, return +* the NID to which it needs to be migrated to. +* +* If not return NUMA_NO_NODE. +*/ +static int kmigrated_get_nid (struct page *page) +{ +struct page_ext *page_ext; +int nid =NUMA_NO_NODE ; + + page_ext =page_ext_get (page); +if (unlikely (!page_ext)) +return nid; + +if (!test_and_clear_bit (PAGE_EXT_MIGRATE_READY , &page_ext->flags)) + goto out; + + nid =page_ext_xchg_nid (page_ext, nid); +out: +page_ext_put (page_ext); +return nid; +} + +/* +* Walks the PFNs of the zone, isolates and migrates them in batches. +*/ +static void kmigrated_walk_zone(unsigned long start_pfn, unsigned long end_pfn,
mm/kmigrated.c line 105
Quoted 10 lines
mm/kmigrated.c new file +10
-0+ int src_nid) +{ +int nid, cur_nid =NUMA_NO_NODE ; +LIST_HEAD (migrate_list); +int batch_count =0 ; +struct folio *folio; +struct page *page; +unsigned long pfn; + +for (pfn = start_pfn; pfn < end_pfn; pfn++) {
Hi,
Is it feasible to scan all the pages in each zone? I think we should figure out a better way so as to reduce CPU time for this purpose.
Besides the opinion above, I was thinking to design and implement a kthread for memory placement between different tiers - I already named it e.g. kmplaced, rather than relying on kswapd and hinting fault, lol ;)
Now that you've started, I'd like to think about it together and improve it so that it works better. Please cc me from the next spin.
Byungchul
Quoted 174 lines
mm/kmigrated.c new file +118
-0+if (!pfn_valid (pfn)) +continue ; + + page =pfn_to_online_page (pfn); +if (!page) +continue ; + +if (page_to_nid (page) != src_nid) +continue ; + +/* +* TODO: Take care of folio_nr_pages() increment +* to pfn count. +*/ + folio =page_folio (page); +if (!folio_test_lru (folio)) +continue ; + + nid =kmigrated_get_nid (page); +if (nid ==NUMA_NO_NODE ) +continue ; + +if (page_to_nid (page) == nid) +continue ; + +if (migrate_misplaced_folio_prepare (folio,NULL , nid)) +continue ; + +if (cur_nid !=NUMA_NO_NODE ) + cur_nid = nid; + +if (++batch_count >=KMIGRATE_BATCH || cur_nid != nid) { +migrate_misplaced_folios_batch (&migrate_list, cur_nid); + cur_nid = nid; + batch_count =0 ; +cond_resched (); + } +list_add (&folio->lru, &migrate_list); + } +if (!list_empty (&migrate_list)) +migrate_misplaced_folios_batch (&migrate_list, cur_nid); +} + +static void kmigrated_do_work (pg_data_t *pgdat) +{ +struct zone *zone; +int zone_idx; + +clear_bit (PGDAT_KMIGRATED_ACTIVATE , &pgdat->flags); +for (zone_idx =0 ; zone_idx <MAX_NR_ZONES ; zone_idx++) { + zone = &pgdat->node_zones[zone_idx]; + +if (!populated_zone (zone)) +continue ; + +if (zone_is_zone_device (zone)) +continue ; + +kmigrated_walk_zone (zone->zone_start_pfn,zone_end_pfn (zone), + pgdat->node_id); + } +} + +static inline bool kmigrated_work_requested (pg_data_t *pgdat) +{ +return test_bit (PGDAT_KMIGRATED_ACTIVATE , &pgdat->flags); +} + +static void kmigrated_wait_work (pg_data_t *pgdat) +{ +long timeout =msecs_to_jiffies (KMIGRATE_DELAY ); + +wait_event_timeout (pgdat->kmigrated_wait, +kmigrated_work_requested (pgdat), timeout); +} + +/* +* Per-node kthread that iterates over its PFNs and migrates the +* pages that have been marked for migration. +*/ +static int kmigrated (void *p) +{ +pg_data_t *pgdat = (pg_data_t *)p; + +while (!kthread_should_stop ()) { +kmigrated_wait_work (pgdat); +kmigrated_do_work (pgdat); + } +return 0 ; +} + +static void kmigrated_run (int nid) +{ +pg_data_t *pgdat =NODE_DATA (nid); + +if (pgdat->kmigrated) +return ; + + pgdat->kmigrated =kthread_create (kmigrated, pgdat,"kmigrated%d" , nid); +if (IS_ERR (pgdat->kmigrated)) { +pr_err ("Failed to start kmigrated for node %d\n" , nid); + pgdat->kmigrated =NULL ; + }else { +wake_up_process (pgdat->kmigrated); + } +} + +static int __init kmigrated_init (void ) +{ +int nid; + +for_each_node_state (nid,N_MEMORY ) +kmigrated_run (nid); + +return 0 ; +} + +subsys_initcall (kmigrated_init)mm/mm_init.c +6
-0@@ -1398,6 +1398,11 @@ static void pgdat_init_kcompactd(struct pglist_data *pgdat)static void pgdat_init_kcompactd (struct pglist_data *pgdat) {}#endif +static void pgdat_init_kmigrated (struct pglist_data *pgdat) +{ +init_waitqueue_head (&pgdat->kmigrated_wait); +} +static void __meminit pgdat_init_internals (struct pglist_data *pgdat) {int i;@@ -1407,6 +1412,7 @@ static void __meminit pgdat_init_internals(struct pglist_data *pgdat)pgdat_init_split_queue (pgdat);pgdat_init_kcompactd (pgdat); +pgdat_init_kmigrated (pgdat);init_waitqueue_head (&pgdat->kswapd_wait);init_waitqueue_head (&pgdat->pfmemalloc_wait);mm/page_ext.c +11
-1@@ -76,6 +76,16 @@ static struct page_ext_operations page_idle_ops __initdata = { };#endif +static bool need_page_mig (void ) +{ +return true; +} + +static struct page_ext_operations page_mig_ops__initdata = { + .need = need_page_mig, + .need_shared_flags = true, +}; +static struct page_ext_operations *page_ext_ops[]__initdata = { #ifdef CONFIG_PAGE_OWNER &page_owner_ops,@@ -89,6 +99,7 @@ static struct page_ext_operations *page_ext_ops[] __initdata = {#ifdef CONFIG_PAGE_TABLE_CHECK &page_table_check_ops, #endif + &page_mig_ops, };unsigned long page_ext_size; --2.34.1
Re: [RFC PATCH v1 3/4] mm: kmigrated - Async kernel migration thread
Quoted 19 lines · Byungchul Park
Quoted 5 lines · Bharata B Rao
mm/kmigrated.c new file +5
-0+ +/* +* Walks the PFNs of the zone, isolates and migrates them in batches. +*/ +static void kmigrated_walk_zone(unsigned long start_pfn, unsigned long end_pfn,mm/kmigrated.c line 105
Quoted 10 lines
mm/kmigrated.c new file +10
-0+ int src_nid) +{ +int nid, cur_nid =NUMA_NO_NODE ; +LIST_HEAD (migrate_list); +int batch_count =0 ; +struct folio *folio; +struct page *page; +unsigned long pfn; + +for (pfn = start_pfn; pfn < end_pfn; pfn++) {Hi,
Quoted 2 lines
Is it feasible to scan all the pages in each zone? I think we should figure out a better way so as to reduce CPU time for this purpose.
I incorporated a per-zone indicator to inform kmigrated if it needs to skip the whole zone when scanning and look at only those zones which have migrate-ready pages.
CPU time spent is one aspect, but the other aspect I have observed is the delay in identifying migrate-ready pages depending on where they exist in the zone. I have been seeing both best case and worst case behaviors due to which the number of pages migrated for a given workload can vary based on the given run.
Hence scanning all pages without additional smarts to quickly arrive the pages of interest may not be ideal. I am working on approaches to improve this situation.
Quoted 7 lines
Besides the opinion above, I was thinking to design and implement a kthread for memory placement between different tiers - I already named it e.g. kmplaced, rather than relying on kswapd and hinting fault, lol ;)
Now that you've started, I'd like to think about it together and improve it so that it works better. Please cc me from the next spin.
Sure, will do from next post.
Regards,
Bharata.
RFC PATCH 4/4 mm: sched: Batch-migrate misplaced pages
In reply to Bharata B RaoCurrently the folios identified as misplaced by the NUMA balancing sub-system are migrated one by one from the NUMA hint fault handler as and when they are identified as misplaced.
Instead of such singe folio migrations, batch them and migrate them at once. This is achieved by passing on the information about misplaced folio to kmigrated which will batch and migrate the folios.
The failed migration count isn't fed back to the scan period update heuristics currently.
--- mm/memory.c | 30 +++++------------------------- 1 file changed, 5 insertions(+), 25 deletions(-)
mm/memory.c +5 -25
@@ -5903,32 +5903,10 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) writable, &last_cpupid);if (target_nid ==NUMA_NO_NODE ) goto out_map; -if (migrate_misplaced_folio_prepare (folio, vma, target_nid)) { - flags |=TNF_MIGRATE_FAIL ; - goto out_map; - } -/* The folio is isolated and isolation code holds a folio reference. */ -pte_unmap_unlock (vmf->pte, vmf->ptl); + writable = false; ignore_writable = true; - -/* Migrate to the requested node */ -if (!migrate_misplaced_folio (folio, target_nid)) { - nid = target_nid; - flags |=TNF_MIGRATED ; -task_numa_fault (last_cpupid, nid, nr_pages, flags); -return 0 ; - } - - flags |=TNF_MIGRATE_FAIL ; - vmf->pte =pte_offset_map_lock (vma->vm_mm, vmf->pmd, - vmf->address, &vmf->ptl); -if (unlikely (!vmf->pte)) -return 0 ; -if (unlikely (!pte_same (ptep_get (vmf->pte), vmf->orig_pte))) { -pte_unmap_unlock (vmf->pte, vmf->ptl); -return 0 ; - } + nid = target_nid; out_map:/* * Make it present again, depending on how arch implements
@@ -5942,8 +5920,10 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) writable);pte_unmap_unlock (vmf->pte, vmf->ptl); -if (nid !=NUMA_NO_NODE ) +if (nid !=NUMA_NO_NODE ) { +kmigrated_add_pfn (folio_pfn (folio), nid);task_numa_fault (last_cpupid, nid, nr_pages, flags); + }return 0 ; }
Signature · 1 line
2.34.1
Huang, Ying
reviewerQuoted 23 lines · Bharata B Rao
Hi,
This is a continuation of the earlier post[1] that attempted to convert migrations from NUMA Balancing to be async and batched. In this version, per-node kernel threads are created to handle migrations in an async manner.
This adds a few fields to the extended page flags that can be used both by the sub-systems that request migrations and kmigrated which migrates the pages. Some of the fields are potentially defined to be used by kpromoted-like subsystem to manage hot page metrics, but are unused right now.
Currently only NUMA Balancing is changed to make use of the async batched migration. It does so by recording the target NID and the readiness of the page to be migrated in the extended page flags fields.
Each kmigrated routinely scans its PFNs, identifies the pages marked for migration and batch-migrates them. Unlike the previous approach, the responsibility of isolating the pages is now with kmigrated.
Quoted 7 lines
The major difference between this approach and the way kpromoted[2] tracked hot pages is the elimination of heavy synchronization points between producers(sub-systems that request migrations or report a hot page) and the consumer (kmigrated or kpromoted). Instead of tracking only the list of hot pages in an orthogonal manner, this approach ties the hot page or migration infomation to the struct page.
I don't think page flag + scanning is a good idea. If the synchronization is really a problem for you (based on test results), some per-CPU data structure can be used to record candidate pages.
[snip]
---
Best Regards,
Huang, Ying
Quoted 4 lines · Ying
Bharata B Rao <[email protected]> writes:
<snip>I don't think page flag + scanning is a good idea.If the
If extended page flags is not the ideal location (I chose it in this version only to get something going quickly), we can look at maintaining per-pfn allocation for the required hot page metadata separately.
Or is your concern specifically with scanning? What problems do you see? It is the cost or the possibility of not identifying the migrate-ready pages in time? Or something else??
Regards,
Bharata.
Huang, Ying
reviewerQuoted 10 lines · Bharata B Rao
Quoted 4 lines · Ying
Bharata B Rao <[email protected]> writes:
<snip>I don't think page flag + scanning is a good idea.If the
If extended page flags is not the ideal location (I chose it in this version only to get something going quickly), we can look at maintaining per-pfn allocation for the required hot page metadata separately.
Quoted 5 lines
Or is your concern specifically with scanning? What problems do you see?
It is the cost or the possibility of not identifying the migrate-ready pages in time? Or something else??
We may need to scan a large number of pages to identify a page to promote. This will waste CPU cycles and pollute cache.
---
Best Regards,
Huang, Ying