Accelerate page migration and use memcg for PMEM management
Subsystems (20)
cover Accelerate page migration and use memcg for PMEM management
From: Zi Yan <[email protected]>
Thanks to Dave Hansen's patches, which make PMEM as part of memory as NUMA nodes. How to use PMEM along with normal DRAM remains an open problem. There are several patchsets posted on the mailing list, proposing to use page migration to move pages between PMEM and DRAM using Linux page replacement policy [1,2,3]. There are some important problems not addressed in these patches:
1. The page migration in Linux does not provide high enough throughput for us to fully exploit PMEM or other use cases.
2. Linux page replacement is running too infrequent to distinguish hot and cold pages.
I am trying to attack the problems with this patch series. This is not a final solution, but I would like to gather more feedback and comments from the mailing list.
Page migration throughput problem
====
For example, in my recent email [4], I gave the page migration throughput numbers for different page migrations, none of which can achieve > 2.5GB/s throughput (the throughput is measured around kernel functions: migrate_pages() and migrate_page_copy()):
| migrate_pages() | migrate_page_copy() migrating single 4KB page: | 0.312GB/s | 1.385GB/s migrating 512 4KB pages: | 0.854GB/s | 1.983GB/s migrating single 2MB THP: | 2.387GB/s | 2.481GB/s
In reality, microbenchmarks show that Intel PMEM can provide ~65GB/s read throughput and ~16GB/s write throughput [5], which are much higher than the throughput achieved by Linux page migration.
In addition, it is also desirable to use page migration to move data between high-bandwidth memory and DRAM, like IBM Summit, which exposes high-performance GPU memories as NUMA nodes [6]. This requires even higher page migration throughput.
In this patch series, I propose four different ways of improving page migration throughput (mostly on 2MB THP migration):
1. multi-threaded page migration: Patch 03 to 06.
2. DMA-based (using Intel IOAT DMA) page migration: Patch 07 and 08.
3. concurrent (batched) page migration: Patch 09, 10, and 11.
4. exchange pages: Patch 12 to 17. (This is a repost of part of [7])
Here are some throughput numbers showing clear throughput improvements on a two-socket NUMA machine with two Xeon E5-2650 v3 @ 2.30GHz and a 19.2GB/s bandwidth QPI link (the same machine as mentioned in [4]):
| migrate_pages() | migrate_page_copy() => migrating single 2MB THP | 2.387GB/s | 2.481GB/s 2-thread single THP migration | 3.478GB/s | 3.704GB/s 4-thread single THP migration | 5.474GB/s | 6.054GB/s 8-thread single THP migration | 7.846GB/s | 9.029GB/s 16-thread single THP migration | 7.423GB/s | 8.464GB/s 16-ch. DMA single THP migration | 4.322GB/s | 4.536GB/s
2-thread 16-THP migration | 3.610GB/s | 3.838GB/s 2-thread 16-THP batched migration | 4.138GB/s | 4.344GB/s 4-thread 16-THP migration | 6.385GB/s | 7.031GB/s 4-thread 16-THP batched migration | 7.382GB/s | 8.072GB/s 8-thread 16-THP migration | 8.039GB/s | 9.029GB/s 8-thread 16-THP batched migration | 9.023GB/s | 10.056GB/s 16-thread 16-THP migration | 8.137GB/s | 9.137GB/s 16-thread 16-THP batched migration | 9.907GB/s | 11.175GB/s
1-thread 16-THP exchange | 4.135GB/s | 4.225GB/s 2-thread 16-THP batched exchange | 7.061GB/s | 7.325GB/s 4-thread 16-THP batched exchange | 9.729GB/s | 10.237GB/s 8-thread 16-THP batched exchange | 9.992GB/s | 10.533GB/s 16-thread 16-THP batched exchange | 9.520GB/s | 10.056GB/s
=> migrating 512 4KB pages | 0.854GB/s | 1.983GB/s 1-thread 512-4KB batched exchange | 1.271GB/s | 3.433GB/s 2-thread 512-4KB batched exchange | 1.240GB/s | 3.190GB/s 4-thread 512-4KB batched exchange | 1.255GB/s | 3.823GB/s 8-thread 512-4KB batched exchange | 1.336GB/s | 3.921GB/s 16-thread 512-4KB batched exchange | 1.334GB/s | 3.897GB/s
Concerns were raised on how to avoid CPU resource competition between page migration and user applications and have power awareness. Daniel Jordan recently posted a multi-threaded ktask patch series could be a solution [8].
Infrequent page list update problem
====
Current page lists are updated by calling shrink_list() when memory pressure comes, which might not be frequent enough to keep track of hot and cold pages. Because all pages are on active lists at the first time shrink_list() is called and the reference bit on the pages might not reflect the up to date access status of these pages. But we also do not want to periodically shrink the global page lists, which adds unnecessary overheads to the whole system. So I propose to actively shrink page lists on the memcg we are interested in.
Patch 18 to 25 add a new system call to shrink page lists on given application's memcg and migrate pages between two NUMA nodes. It isolates the impact from the rest of the system. To share DRAM among different applications, Patch 18 and 19 add per-node memcg size limit, so you can limit the memory usage for particular NUMA node(s).
Patch structure
====
1. multi-threaded page migration: Patch 01 to 06.
2. DMA-based (using Intel IOAT DMA) page migration: Patch 07 and 08.
3. concurrent (batched) page migration: Patch 09, 10, and 11.
4. exchange pages: Patch 12 to 17. (This is a repost of part of [7])
5. per-node size limit in memcg: Patch 18 and 19.
6. actively shrink page lists and perform page migration in given memcg: Patch 20 to 25.
Any comment is welcome.
[1]: https://lore.kernel.org/linux-mm/[email protected]/
[2]: https://lore.kernel.org/linux-mm/[email protected]/
[3]: https://lore.kernel.org/linux-mm/[email protected]/
[4]: https://lore.kernel.org/linux-mm/[email protected]/
[5]: https://www.storagereview.com/supermicro_superserver_with_intel_optane_dc_persistent_memory_first_look_review
[6]: https://www.ibm.com/thought-leadership/summit-supercomputer/
[7]: https://lore.kernel.org/linux-mm/[email protected]/
[8]: https://lore.kernel.org/linux-mm/[email protected]/
Zi Yan (25):
mm: migrate: Change migrate_mode to support combination migration
modes.
mm: migrate: Add mode parameter to support future page copy routines.
mm: migrate: Add a multi-threaded page migration function.
mm: migrate: Add copy_page_multithread into migrate_pages.
mm: migrate: Add vm.accel_page_copy in sysfs to control page copy
acceleration.
mm: migrate: Make the number of copy threads adjustable via sysctl.
mm: migrate: Add copy_page_dma to use DMA Engine to copy pages.
mm: migrate: Add copy_page_dma into migrate_page_copy.
mm: migrate: Add copy_page_lists_dma_always to support copy a list of
pages.
mm: migrate: copy_page_lists_mt() to copy a page list using
multi-threads.
mm: migrate: Add concurrent page migration into move_pages syscall. exchange pages: new page migration mechanism: exchange_pages() exchange pages: add multi-threaded exchange pages. exchange pages: concurrent exchange pages.
exchange pages: exchange anonymous page and file-backed page. exchange page: Add THP exchange support.
exchange page: Add exchange_page() syscall.
memcg: Add per node memory usage&max stats in memcg.
mempolicy: add MPOL_F_MEMCG flag, enforcing memcg memory limit. memory manage: Add memory manage syscall.
mm: move update_lru_sizes() to mm_inline.h for broader use. memory manage: active/inactive page list manipulation in memcg. memory manage: page migration based page manipulation between NUMA
nodes.
memory manage: limit migration batch size.
memory manage: use exchange pages to memory manage to improve
throughput.
arch/x86/entry/syscalls/syscall_64.tbl | 2 + fs/aio.c | 12 +- fs/f2fs/data.c | 6 +- fs/hugetlbfs/inode.c | 4 +- fs/iomap.c | 4 +- fs/ubifs/file.c | 4 +- include/linux/cgroup-defs.h | 1 + include/linux/exchange.h | 27 + include/linux/highmem.h | 3 + include/linux/ksm.h | 4 + include/linux/memcontrol.h | 67 ++ include/linux/migrate.h | 12 +- include/linux/migrate_mode.h | 8 + include/linux/mm_inline.h | 21 + include/linux/sched/coredump.h | 1 + include/linux/sched/sysctl.h | 3 + include/linux/syscalls.h | 10 + include/uapi/linux/mempolicy.h | 9 +- kernel/sysctl.c | 47 + mm/Makefile | 5 + mm/balloon_compaction.c | 2 +- mm/compaction.c | 22 +- mm/copy_page.c | 708 +++++++++++++++ mm/exchange.c | 1560 ++++++++++++++++++++++++++++++++ mm/exchange_page.c | 228 +++++ mm/internal.h | 113 +++ mm/ksm.c | 35 + mm/memcontrol.c | 80 ++ mm/memory_manage.c | 649 +++++++++++++ mm/mempolicy.c | 38 +- mm/migrate.c | 621 ++++++++++++- mm/vmscan.c | 115 +-- mm/zsmalloc.c | 2 +- 33 files changed, 4261 insertions(+), 162 deletions(-) create mode 100644 include/linux/exchange.h create mode 100644 mm/copy_page.c create mode 100644 mm/exchange.c create mode 100644 mm/exchange_page.c create mode 100644 mm/memory_manage.c
--
2.7.4
RFC PATCH 1/25 mm: migrate: Change migrate_mode to support combination migration modes.
From: Zi Yan <[email protected]>
No functionality is changed. Prepare for the following patches, which add parallel, concurrent page migration modes in conjunction to the existing modes.
--- fs/aio.c | 10 +++++----- fs/f2fs/data.c | 4 ++-- fs/hugetlbfs/inode.c | 2 +- fs/iomap.c | 2 +- fs/ubifs/file.c | 2 +- include/linux/migrate_mode.h | 2 ++ mm/balloon_compaction.c | 2 +- mm/compaction.c | 22 +++++++++++----------- mm/migrate.c | 18 +++++++++--------- mm/zsmalloc.c | 2 +- 10 files changed, 34 insertions(+), 32 deletions(-)
fs/aio.c +5 -5
@@ -389,7 +389,7 @@ static int aio_migratepage(struct address_space *mapping, struct page *new,* happen under the ctx->completion_lock. That does not work with the * migration workflow of MIGRATE_SYNC_NO_COPY. */ -if (mode ==MIGRATE_SYNC_NO_COPY ) +if ((mode &MIGRATE_MODE_MASK ) ==MIGRATE_SYNC_NO_COPY )return -EINVAL ; rc =0 ;
@@ -1300,10 +1300,10 @@ static long read_events(struct kioctx *ctx, long min_nr, long nr,* Create an aio_context capable of receiving at least nr_events. * ctxp must not point to an aio_context that already exists, and * must be initialized to 0 prior to the call. On successful -* creation of the aio_context, *ctxp is filled in with the resulting +* creation of the aio_context, *ctxp is filled in with the resulting * handle. May fail with -EINVAL if *ctxp is not initialized, -* if the specified nr_events exceeds internal limits. May fail -* with -EAGAIN if the specified nr_events exceeds the user's limit +* if the specified nr_events exceeds internal limits. May fail +* with -EAGAIN if the specified nr_events exceeds the user's limit * of available events. May fail with -ENOMEM if insufficient kernel * resources are available. May fail with -EFAULT if an invalid * pointer is passed for ctxp. Will fail with -ENOSYS if not
@@ -1373,7 +1373,7 @@ COMPAT_SYSCALL_DEFINE2(io_setup, unsigned, nr_events, u32 __user *, ctx32p)#endif /* sys_io_destroy: -* Destroy the aio_context specified. May cancel any outstanding +* Destroy the aio_context specified. May cancel any outstanding * AIOs and block on completion. Will fail with -ENOSYS if not * implemented. May fail with -EINVAL if the context pointed to * is invalid.
fs/f2fs/data.c +2 -2
@@ -2792,7 +2792,7 @@ int f2fs_migrate_page(struct address_space *mapping,/* migrating an atomic written page is safe with the inmem_lock hold */ if (atomic_written) { -if (mode !=MIGRATE_SYNC ) +if ((mode &MIGRATE_MODE_MASK ) !=MIGRATE_SYNC )return -EBUSY ;if (!mutex_trylock (&fi->inmem_lock))return -EAGAIN ;
@@ -2825,7 +2825,7 @@ int f2fs_migrate_page(struct address_space *mapping,f2fs_clear_page_private (page); } -if (mode !=MIGRATE_SYNC_NO_COPY ) +if ((mode &MIGRATE_MODE_MASK ) !=MIGRATE_SYNC_NO_COPY )migrate_page_copy (newpage, page);else migrate_page_states (newpage, page);
fs/hugetlbfs/inode.c +1 -1
@@ -885,7 +885,7 @@ static int hugetlbfs_migrate_page(struct address_space *mapping,set_page_private (page,0 ); } -if (mode !=MIGRATE_SYNC_NO_COPY ) +if ((mode &MIGRATE_MODE_MASK ) !=MIGRATE_SYNC_NO_COPY )migrate_page_copy (newpage, page);else migrate_page_states (newpage, page);
fs/iomap.c +1 -1
@@ -584,7 +584,7 @@ iomap_migrate_page(struct address_space *mapping, struct page *newpage,SetPagePrivate (newpage); } -if (mode !=MIGRATE_SYNC_NO_COPY ) +if ((mode &MIGRATE_MODE_MASK ) !=MIGRATE_SYNC_NO_COPY )migrate_page_copy (newpage, page);else migrate_page_states (newpage, page);
fs/ubifs/file.c +1 -1
@@ -1490,7 +1490,7 @@ static int ubifs_migrate_page(struct address_space *mapping,SetPagePrivate (newpage); } -if (mode !=MIGRATE_SYNC_NO_COPY ) +if ((mode &MIGRATE_MODE_MASK ) !=MIGRATE_SYNC_NO_COPY )migrate_page_copy (newpage, page);else migrate_page_states (newpage, page);
include/linux/migrate_mode.h +2 -0
@@ -17,6 +17,8 @@ enum migrate_mode {MIGRATE_SYNC_LIGHT ,MIGRATE_SYNC ,MIGRATE_SYNC_NO_COPY , + + MIGRATE_MODE_MASK = 3, };#endif /* MIGRATE_MODE_H_INCLUDED */
mm/balloon_compaction.c +1 -1
@@ -158,7 +158,7 @@ int balloon_page_migrate(struct address_space *mapping,* is unlikely to be use with ballon pages. See include/linux/hmm.h for * user of the MIGRATE_SYNC_NO_COPY mode. */ -if (mode ==MIGRATE_SYNC_NO_COPY ) +if ((mode &MIGRATE_MODE_MASK ) ==MIGRATE_SYNC_NO_COPY )return -EINVAL ;VM_BUG_ON_PAGE (!PageLocked (page), page);
mm/compaction.c +11 -11
@@ -408,7 +408,7 @@ static void update_cached_migrate(struct compact_control *cc, unsigned long pfn)if (pfn > zone->compact_cached_migrate_pfn[0 ]) zone->compact_cached_migrate_pfn[0 ] = pfn; -if (cc->mode !=MIGRATE_ASYNC && +if ((cc->mode &MIGRATE_MODE_MASK ) !=MIGRATE_ASYNC && pfn > zone->compact_cached_migrate_pfn[1 ]) zone->compact_cached_migrate_pfn[1 ] = pfn; }
@@ -475,7 +475,7 @@ static bool compact_lock_irqsave(spinlock_t *lock, unsigned long *flags, struct compact_control *cc) {/* Track if the lock is contended in async mode */ -if (cc->mode ==MIGRATE_ASYNC && !cc->contended) { +if (((cc->mode &MIGRATE_MODE_MASK ) ==MIGRATE_ASYNC ) && !cc->contended) {if (spin_trylock_irqsave (lock, *flags))return true;
@@ -792,7 +792,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,*/ while (unlikely (too_many_isolated (pgdat))) {/* async migration should just abort */ -if (cc->mode ==MIGRATE_ASYNC ) +if ((cc->mode &MIGRATE_MODE_MASK ) ==MIGRATE_ASYNC )return 0 ;congestion_wait (BLK_RW_ASYNC ,HZ /10 );
@@ -803,7 +803,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,cond_resched (); -if (cc->direct_compaction && (cc->mode ==MIGRATE_ASYNC )) { +if (cc->direct_compaction && ((cc->mode &MIGRATE_MODE_MASK ) ==MIGRATE_ASYNC )) { skip_on_failure = true; next_skip_pfn =block_end_pfn (low_pfn, cc->order); }
@@ -1117,7 +1117,7 @@ static bool suitable_migration_source(struct compact_control *cc,if (pageblock_skip_persistent (page))return false; -if ((cc->mode !=MIGRATE_ASYNC ) || !cc->direct_compaction) +if (((cc->mode &MIGRATE_MODE_MASK ) !=MIGRATE_ASYNC ) || !cc->direct_compaction)return true; block_mt =get_pageblock_migratetype (page);
@@ -1216,7 +1216,7 @@ fast_isolate_around(struct compact_control *cc, unsigned long pfn, unsigned longreturn ;/* Minimise scanning during async compaction */ -if (cc->direct_compaction && cc->mode ==MIGRATE_ASYNC ) +if (cc->direct_compaction && (cc->mode &MIGRATE_MODE_MASK ) ==MIGRATE_ASYNC )return ;/* Pageblock boundaries */
@@ -1448,7 +1448,7 @@ static void isolate_freepages(struct compact_control *cc) block_end_pfn =min (block_start_pfn + pageblock_nr_pages,zone_end_pfn (zone)); low_pfn =pageblock_end_pfn (cc->migrate_pfn); - stride = cc->mode ==MIGRATE_ASYNC ?COMPACT_CLUSTER_MAX :1 ; + stride = (cc->mode &MIGRATE_MODE_MASK ) ==MIGRATE_ASYNC ?COMPACT_CLUSTER_MAX :1 ;/* * Isolate free pages until enough are available to migrate the
@@ -1734,7 +1734,7 @@ static isolate_migrate_t isolate_migratepages(struct zone *zone,struct page *page;const isolate_mode_t isolate_mode = (sysctl_compact_unevictable_allowed ?ISOLATE_UNEVICTABLE :0 ) | - (cc->mode !=MIGRATE_SYNC ?ISOLATE_ASYNC_MIGRATE :0 ); + (((cc->mode &MIGRATE_MODE_MASK ) !=MIGRATE_SYNC ) ?ISOLATE_ASYNC_MIGRATE :0 );bool fast_find_block;/*
@@ -1907,7 +1907,7 @@ static enum compact_result __compact_finished(struct compact_control *cc)* to sync compaction, as async compaction operates * on pageblocks of the same migratetype. */ -if (cc->mode ==MIGRATE_ASYNC || +if ((cc->mode &MIGRATE_MODE_MASK ) ==MIGRATE_ASYNC ||IS_ALIGNED (cc->migrate_pfn, pageblock_nr_pages)) {return COMPACT_SUCCESS ;
@@ -2063,7 +2063,7 @@ compact_zone(struct compact_control *cc, struct capture_control *capc)unsigned long start_pfn = cc->zone->zone_start_pfn;unsigned long end_pfn =zone_end_pfn (cc->zone);unsigned long last_migrated_pfn; -const bool sync = cc->mode !=MIGRATE_ASYNC ; +const bool sync = (cc->mode &MIGRATE_MODE_MASK ) !=MIGRATE_ASYNC ;bool update_cached; cc->migratetype =gfpflags_to_migratetype (cc->gfp_mask);
@@ -2195,7 +2195,7 @@ compact_zone(struct compact_control *cc, struct capture_control *capc)* order-aligned block, so skip the rest of it. */ if (cc->direct_compaction && - (cc->mode ==MIGRATE_ASYNC )) { + ((cc->mode &MIGRATE_MODE_MASK ) ==MIGRATE_ASYNC )) { cc->migrate_pfn =block_end_pfn ( cc->migrate_pfn -1 , cc->order);/* Draining pcplists is useless in this case */
mm/migrate.c +9 -9
@@ -691,7 +691,7 @@ int migrate_page(struct address_space *mapping,if (rc !=MIGRATEPAGE_SUCCESS )return rc; -if (mode !=MIGRATE_SYNC_NO_COPY ) +if ((mode &MIGRATE_MODE_MASK ) !=MIGRATE_SYNC_NO_COPY )migrate_page_copy (newpage, page);else migrate_page_states (newpage, page);
@@ -707,7 +707,7 @@ static bool buffer_migrate_lock_buffers(struct buffer_head *head,struct buffer_head *bh = head;/* Simple case, sync compaction */ -if (mode !=MIGRATE_ASYNC ) { +if ((mode &MIGRATE_MODE_MASK ) !=MIGRATE_ASYNC ) { do { lock_buffer(bh); bh = bh->b_this_page;
@@ -804,7 +804,7 @@ static int __buffer_migrate_page(struct address_space *mapping,SetPagePrivate (newpage); -if (mode !=MIGRATE_SYNC_NO_COPY ) +if ((mode &MIGRATE_MODE_MASK ) !=MIGRATE_SYNC_NO_COPY )migrate_page_copy (newpage, page);else migrate_page_states (newpage, page);
@@ -895,7 +895,7 @@ static int fallback_migrate_page(struct address_space *mapping, {if (PageDirty (page)) {/* Only writeback pages in full synchronous migration */ -switch (mode) { +switch (mode &MIGRATE_MODE_MASK ) {case MIGRATE_SYNC :case MIGRATE_SYNC_NO_COPY :break ;
@@ -911,7 +911,7 @@ static int fallback_migrate_page(struct address_space *mapping,*/ if (page_has_private (page) && !try_to_release_page (page,GFP_KERNEL )) -return mode ==MIGRATE_SYNC ? -EAGAIN : -EBUSY ; +return (mode &MIGRATE_MODE_MASK ) ==MIGRATE_SYNC ? -EAGAIN : -EBUSY ;return migrate_page (mapping, newpage, page, mode); }
@@ -1009,7 +1009,7 @@ static int __unmap_and_move(struct page *page, struct page *newpage,bool is_lru = !__PageMovable (page);if (!trylock_page (page)) { -if (!force || mode ==MIGRATE_ASYNC ) +if (!force || ((mode &MIGRATE_MODE_MASK ) ==MIGRATE_ASYNC )) goto out;/*
@@ -1038,7 +1038,7 @@ static int __unmap_and_move(struct page *page, struct page *newpage,* the retry loop is too short and in the sync-light case, * the overhead of stalling is too much */ -switch (mode) { +switch (mode &MIGRATE_MODE_MASK ) {case MIGRATE_SYNC :case MIGRATE_SYNC_NO_COPY :break ;
@@ -1303,9 +1303,9 @@ static int unmap_and_move_huge_page(new_page_t get_new_page,return -ENOMEM ;if (!trylock_page (hpage)) { -if (!force) +if (!force || ((mode &MIGRATE_MODE_MASK ) !=MIGRATE_SYNC )) goto out; -switch (mode) { +switch (mode &MIGRATE_MODE_MASK ) {case MIGRATE_SYNC :case MIGRATE_SYNC_NO_COPY :break ;
mm/zsmalloc.c +1 -1
@@ -1981,7 +1981,7 @@ static int zs_page_migrate(struct address_space *mapping, struct page *newpage,* happen under the zs lock, which does not work with * MIGRATE_SYNC_NO_COPY workflow. */ -if (mode ==MIGRATE_SYNC_NO_COPY ) +if ((mode &MIGRATE_MODE_MASK ) ==MIGRATE_SYNC_NO_COPY )return -EINVAL ;VM_BUG_ON_PAGE (!PageMovable (page), page);
Signature · 1 line
2.7.4
RFC PATCH 2/25 mm: migrate: Add mode parameter to support future page copy routines.
In reply to Zi YanFrom: Zi Yan <[email protected]>
MIGRATE_SINGLETHREAD is added as the default behavior. migrate_page_copy() and copy_huge_page() are changed.
--- fs/aio.c | 2 +- fs/f2fs/data.c | 2 +- fs/hugetlbfs/inode.c | 2 +- fs/iomap.c | 2 +- fs/ubifs/file.c | 2 +- include/linux/migrate.h | 6 ++++-- include/linux/migrate_mode.h | 3 +++ mm/migrate.c | 14 ++++++++------ 8 files changed, 20 insertions(+), 13 deletions(-)
fs/aio.c +1 -1
@@ -437,7 +437,7 @@ static int aio_migratepage(struct address_space *mapping, struct page *new,* events from being lost. */ spin_lock_irqsave (&ctx->completion_lock, flags); -migrate_page_copy (new, old); +migrate_page_copy (new, old,MIGRATE_SINGLETHREAD );BUG_ON (ctx->ring_pages[idx] != old); ctx->ring_pages[idx] = new;spin_unlock_irqrestore (&ctx->completion_lock, flags);
fs/f2fs/data.c +1 -1
@@ -2826,7 +2826,7 @@ int f2fs_migrate_page(struct address_space *mapping, }if ((mode &MIGRATE_MODE_MASK ) !=MIGRATE_SYNC_NO_COPY ) -migrate_page_copy (newpage, page); +migrate_page_copy (newpage, page,MIGRATE_SINGLETHREAD );else migrate_page_states (newpage, page);
fs/hugetlbfs/inode.c +1 -1
@@ -886,7 +886,7 @@ static int hugetlbfs_migrate_page(struct address_space *mapping, }if ((mode &MIGRATE_MODE_MASK ) !=MIGRATE_SYNC_NO_COPY ) -migrate_page_copy (newpage, page); +migrate_page_copy (newpage, page,MIGRATE_SINGLETHREAD );else migrate_page_states (newpage, page);
fs/iomap.c +1 -1
@@ -585,7 +585,7 @@ iomap_migrate_page(struct address_space *mapping, struct page *newpage, }if ((mode &MIGRATE_MODE_MASK ) !=MIGRATE_SYNC_NO_COPY ) -migrate_page_copy (newpage, page); +migrate_page_copy (newpage, page,MIGRATE_SINGLETHREAD );else migrate_page_states (newpage, page);return MIGRATEPAGE_SUCCESS ;
fs/ubifs/file.c +1 -1
@@ -1491,7 +1491,7 @@ static int ubifs_migrate_page(struct address_space *mapping, }if ((mode &MIGRATE_MODE_MASK ) !=MIGRATE_SYNC_NO_COPY ) -migrate_page_copy (newpage, page); +migrate_page_copy (newpage, page,MIGRATE_SINGLETHREAD );else migrate_page_states (newpage, page);return MIGRATEPAGE_SUCCESS ;
include/linux/migrate.h +4 -2
@@ -73,7 +73,8 @@ extern void putback_movable_page(struct page *page);extern int migrate_prep (void );extern int migrate_prep_local (void );extern void migrate_page_states (struct page *newpage,struct page *page); -extern void migrate_page_copy (struct page *newpage,struct page *page); +extern void migrate_page_copy (struct page *newpage,struct page *page, +enum migrate_mode mode);extern int migrate_huge_page_move_mapping (struct address_space *mapping,struct page *newpage,struct page *page); extern int migrate_page_move_mapping(struct address_space *mapping,
@@ -97,7 +98,8 @@ static inline void migrate_page_states(struct page *newpage, struct page *page) }static inline void migrate_page_copy (struct page *newpage, -struct page *page) {} +struct page *page, +enum migrate_mode mode) {}static inline int migrate_huge_page_move_mapping (struct address_space *mapping,struct page *newpage,struct page *page)
include/linux/migrate_mode.h +3 -0
@@ -11,6 +11,8 @@* with the CPU. Instead, page copy happens outside the migratepage() * callback and is likely using a DMA engine. See migrate_vma() and HMM * (mm/hmm.c) for users of this mode. +* MIGRATE_SINGLETHREAD uses a single thread to move pages, it is the default +* behavior */ enum migrate_mode {MIGRATE_ASYNC ,
@@ -19,6 +21,7 @@ enum migrate_mode {MIGRATE_SYNC_NO_COPY ,MIGRATE_MODE_MASK =3 , + MIGRATE_SINGLETHREAD = 0, };#endif /* MIGRATE_MODE_H_INCLUDED */
mm/migrate.c +8 -6
@@ -567,7 +567,8 @@ static void __copy_gigantic_page(struct page *dst, struct page *src, } } -static void copy_huge_page (struct page *dst,struct page *src) +static void copy_huge_page (struct page *dst,struct page *src, +enum migrate_mode mode) {int i;int nr_pages;
@@ -657,10 +658,11 @@ void migrate_page_states(struct page *newpage, struct page *page) }EXPORT_SYMBOL (migrate_page_states); -void migrate_page_copy (struct page *newpage,struct page *page) +void migrate_page_copy (struct page *newpage,struct page *page, +enum migrate_mode mode) {if (PageHuge (page) ||PageTransHuge (page)) -copy_huge_page (newpage, page); +copy_huge_page (newpage, page, mode);else copy_highpage (newpage, page);
@@ -692,7 +694,7 @@ int migrate_page(struct address_space *mapping,return rc;if ((mode &MIGRATE_MODE_MASK ) !=MIGRATE_SYNC_NO_COPY ) -migrate_page_copy (newpage, page); +migrate_page_copy (newpage, page, mode);else migrate_page_states (newpage, page);return MIGRATEPAGE_SUCCESS ;
@@ -805,7 +807,7 @@ static int __buffer_migrate_page(struct address_space *mapping,SetPagePrivate (newpage);if ((mode &MIGRATE_MODE_MASK ) !=MIGRATE_SYNC_NO_COPY ) -migrate_page_copy (newpage, page); +migrate_page_copy (newpage, page,MIGRATE_SINGLETHREAD );else migrate_page_states (newpage, page);
@@ -2024,7 +2026,7 @@ int migrate_misplaced_transhuge_page(struct mm_struct *mm, new_page->index = page->index;/* flush the cache before copying using the kernel virtual address */ flush_cache_range (vma, start, start +HPAGE_PMD_SIZE ); -migrate_page_copy (new_page, page); +migrate_page_copy (new_page, page,MIGRATE_SINGLETHREAD );WARN_ON (PageLRU (new_page));/* Recheck the target PMD */
Signature · 1 line
2.7.4
RFC PATCH 3/25 mm: migrate: Add a multi-threaded page migration function.
In reply to Zi YanFrom: Zi Yan <[email protected]>
copy_page_multithread() function is added to migrate huge pages in multi-threaded way, which provides higher throughput than a single-threaded way.
Internally, copy_page_multithread() splits and distributes a huge page into multiple threads, then send them as jobs to system_highpri_wq.
--- include/linux/highmem.h | 2 + mm/Makefile | 2 + mm/copy_page.c | 128 ++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 132 insertions(+) create mode 100644 mm/copy_page.c
include/linux/highmem.h +2 -0
@@ -276,4 +276,6 @@ static inline void copy_highpage(struct page *to, struct page *from)#endif +int copy_page_multithread (struct page *to,struct page *from,int nr_pages); +#endif /* _LINUX_HIGHMEM_H */
mm/Makefile +2 -0
@@ -44,6 +44,8 @@ obj-y := filemap.o mempool.o oom_kill.o fadvise.o \ obj-y += init-mm.o obj-y += memblock.o +obj-y += copy_page.o + ifdef CONFIG_MMU obj-$(CONFIG_ADVISE_SYSCALLS) += madvise.o endif
mm/copy_page.c new file +128 -0
128 lines not on this page · raw message
Signature · 1 line
2.7.4
RFC PATCH 4/25 mm: migrate: Add copy_page_multithread into migrate_pages.
In reply to Zi YanFrom: Zi Yan <[email protected]>
An option is added to move_pages() syscall to use multi-threaded page migration.
--- include/linux/migrate_mode.h | 1 + include/uapi/linux/mempolicy.h | 2 ++ mm/migrate.c | 29 +++++++++++++++++++---------- 3 files changed, 22 insertions(+), 10 deletions(-)
include/linux/migrate_mode.h +1 -0
@@ -22,6 +22,7 @@ enum migrate_mode {MIGRATE_MODE_MASK =3 ,MIGRATE_SINGLETHREAD =0 , + MIGRATE_MT = 1<<4, };#endif /* MIGRATE_MODE_H_INCLUDED */
include/uapi/linux/mempolicy.h +2 -0
@@ -48,6 +48,8 @@ enum {#define MPOL_MF_LAZY (1<<3)/* Modifies '_MOVE: lazy migrate on fault */ #define MPOL_MF_INTERNAL (1<<4)/* Internal flags start here */ +#define MPOL_MF_MOVE_MT (1<<6)/* Use multi-threaded page copy routine */ +#define MPOL_MF_VALID (MPOL_MF_STRICT | \ MPOL_MF_MOVE | \ MPOL_MF_MOVE_ALL)
mm/migrate.c +19 -10
@@ -572,6 +572,7 @@ static void copy_huge_page(struct page *dst, struct page *src, {int i;int nr_pages; +int rc = -EFAULT ;if (PageHuge (src)) {/* hugetlbfs page */
@@ -588,10 +589,14 @@ static void copy_huge_page(struct page *dst, struct page *src, nr_pages =hpage_nr_pages (src); } -for (i =0 ; i < nr_pages; i++) { -cond_resched (); -copy_highpage (dst + i, src + i); - } +if (mode &MIGRATE_MT ) + rc =copy_page_multithread (dst, src, nr_pages); + +if (rc) +for (i =0 ; i < nr_pages; i++) { +cond_resched (); +copy_highpage (dst + i, src + i); + } }/*
@@ -1500,7 +1505,7 @@ static int store_status(int __user *status, int start, int value, int nr) }static int do_move_pages_to_node (struct mm_struct *mm, -struct list_head *pagelist,int node) +struct list_head *pagelist,int node,bool migrate_mt) {int err;
@@ -1508,7 +1513,8 @@ static int do_move_pages_to_node(struct mm_struct *mm,return 0 ; err =migrate_pages (pagelist, alloc_new_node_page,NULL , node, -MIGRATE_SYNC ,MR_SYSCALL ); +MIGRATE_SYNC | (migrate_mt ?MIGRATE_MT :MIGRATE_SINGLETHREAD ), +MR_SYSCALL );if (err)putback_movable_pages (pagelist);return err;
@@ -1629,7 +1635,8 @@ static int do_pages_move(struct mm_struct *mm, nodemask_t task_nodes, current_node = node; start = i; }else if (node != current_node) { - err =do_move_pages_to_node (mm, &pagelist, current_node); + err =do_move_pages_to_node (mm, &pagelist, current_node, + flags &MPOL_MF_MOVE_MT );if (err) goto out; err =store_status (status, start, current_node, i - start);
@@ -1652,7 +1659,8 @@ static int do_pages_move(struct mm_struct *mm, nodemask_t task_nodes,if (err) goto out_flush; - err =do_move_pages_to_node (mm, &pagelist, current_node); + err =do_move_pages_to_node (mm, &pagelist, current_node, + flags &MPOL_MF_MOVE_MT );if (err) goto out;if (i > start) {
@@ -1667,7 +1675,8 @@ static int do_pages_move(struct mm_struct *mm, nodemask_t task_nodes,return err;/* Make sure we do not overwrite the existing error */ - err1 =do_move_pages_to_node (mm, &pagelist, current_node); + err1 =do_move_pages_to_node (mm, &pagelist, current_node, + flags &MPOL_MF_MOVE_MT );if (!err1) err1 =store_status (status, start, current_node, i - start);if (!err)
@@ -1763,7 +1772,7 @@ static int kernel_move_pages(pid_t pid, unsigned long nr_pages,nodemask_t task_nodes;/* Check flags */ -if (flags & ~(MPOL_MF_MOVE |MPOL_MF_MOVE_ALL )) +if (flags & ~(MPOL_MF_MOVE |MPOL_MF_MOVE_ALL |MPOL_MF_MOVE_MT ))return -EINVAL ;if ((flags &MPOL_MF_MOVE_ALL ) && !capable (CAP_SYS_NICE ))
Signature · 1 line
2.7.4
RFC PATCH 5/25 mm: migrate: Add vm.accel_page_copy in sysfs to control page copy acceleration.
In reply to Zi YanFrom: Zi Yan <[email protected]>
Since base page migration did not gain any speedup from multi-threaded methods, we only accelerate the huge page case.
--- kernel/sysctl.c | 11 +++++++++++ mm/migrate.c | 6 ++++++ 2 files changed, 17 insertions(+)
kernel/sysctl.c +11 -0
@@ -101,6 +101,8 @@#if defined(CONFIG_SYSCTL ) +extern int accel_page_copy; +/* External variables not in a header file. */ extern int suid_dumpable;#ifdef CONFIG_COREDUMP
@@ -1430,6 +1432,15 @@ static struct ctl_table vm_table[] = { .extra2 = &one, },#endif + { + .procname = "accel_page_copy", + .data = &accel_page_copy, + .maxlen = sizeof(accel_page_copy), + .mode = 0644, + .proc_handler = proc_dointvec, + .extra1 = &zero, + .extra2 = &one, + }, { .procname = "hugetlb_shm_group", .data = &sysctl_hugetlb_shm_group,
mm/migrate.c +6 -0
@@ -55,6 +55,8 @@#include "internal.h" +int accel_page_copy =1 ; +/* * migrate_prep() needs to be called before we start compiling a list of pages * to be migrated using isolate_lru_page(). If scheduling work on other CPUs is
@@ -589,6 +591,10 @@ static void copy_huge_page(struct page *dst, struct page *src, nr_pages =hpage_nr_pages (src); } +/* Try to accelerate page migration if it is not specified in mode */ +if (accel_page_copy) + mode |=MIGRATE_MT ; +if (mode &MIGRATE_MT ) rc =copy_page_multithread (dst, src, nr_pages);
Signature · 1 line
2.7.4
RFC PATCH 6/25 mm: migrate: Make the number of copy threads adjustable via sysctl.
In reply to Zi YanFrom: Zi Yan <[email protected]>
--- kernel/sysctl.c | 9 +++++++++ mm/copy_page.c | 2 +- 2 files changed, 10 insertions(+), 1 deletion(-)
kernel/sysctl.c +9 -0
@@ -102,6 +102,7 @@#if defined(CONFIG_SYSCTL )extern int accel_page_copy; +extern unsigned int limit_mt_num;/* External variables not in a header file. */ extern int suid_dumpable;
@@ -1441,6 +1442,14 @@ static struct ctl_table vm_table[] = { .extra1 = &zero, .extra2 = &one, }, + { + .procname = "limit_mt_num", + .data = &limit_mt_num, + .maxlen = sizeof(limit_mt_num), + .mode = 0644, + .proc_handler = proc_dointvec, + .extra1 = &zero, + }, { .procname = "hugetlb_shm_group", .data = &sysctl_hugetlb_shm_group,
mm/copy_page.c +1 -1
@@ -23,7 +23,7 @@#include <linux/freezer.h> -const unsigned int limit_mt_num =4 ; +unsigned int limit_mt_num =4 ;/* ======================== multi-threaded copy page ======================== */
Signature · 1 line
2.7.4
RFC PATCH 7/25 mm: migrate: Add copy_page_dma to use DMA Engine to copy pages.
In reply to Zi YanFrom: Zi Yan <[email protected]>
vm.use_all_dma_chans will grab all usable DMA channels vm.limit_dma_chans will limit how many DMA channels in use
--- include/linux/highmem.h | 1 + include/linux/sched/sysctl.h | 3 + kernel/sysctl.c | 19 +++ mm/copy_page.c | 291 +++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 314 insertions(+)
include/linux/highmem.h +1 -0
@@ -277,5 +277,6 @@ static inline void copy_highpage(struct page *to, struct page *from)#endif int copy_page_multithread (struct page *to,struct page *from,int nr_pages); +int copy_page_dma (struct page *to,struct page *from,int nr_pages);#endif /* _LINUX_HIGHMEM_H */
include/linux/sched/sysctl.h +3 -0
@@ -90,4 +90,7 @@ extern int sched_energy_aware_handler(struct ctl_table *table, int write, loff_t *ppos);#endif +extern int sysctl_dma_page_migration (struct ctl_table *table,int write, +void __user *buffer,size_t *lenp, +loff_t *ppos);#endif /* _LINUX_SCHED_SYSCTL_H */
kernel/sysctl.c +19 -0
@@ -103,6 +103,8 @@extern int accel_page_copy;extern unsigned int limit_mt_num; +extern int use_all_dma_chans; +extern int limit_dma_chans;/* External variables not in a header file. */ extern int suid_dumpable;
@@ -1451,6 +1453,23 @@ static struct ctl_table vm_table[] = { .extra1 = &zero, }, { + .procname = "use_all_dma_chans", + .data = &use_all_dma_chans, + .maxlen = sizeof(use_all_dma_chans), + .mode = 0644, + .proc_handler = sysctl_dma_page_migration, + .extra1 = &zero, + .extra2 = &one, + }, + { + .procname = "limit_dma_chans", + .data = &limit_dma_chans, + .maxlen = sizeof(limit_dma_chans), + .mode = 0644, + .proc_handler = proc_dointvec, + .extra1 = &zero, + }, + { .procname = "hugetlb_shm_group", .data = &sysctl_hugetlb_shm_group, .maxlen = sizeof(gid_t),
mm/copy_page.c +291 -0
294 lines not on this page · raw message
Signature · 1 line
2.7.4
RFC PATCH 8/25 mm: migrate: Add copy_page_dma into migrate_page_copy.
In reply to Zi YanFrom: Zi Yan <[email protected]>
Fallback to copy_highpage when it fails.
--- include/linux/migrate_mode.h | 1 + include/uapi/linux/mempolicy.h | 1 + mm/migrate.c | 31 +++++++++++++++++++++---------- 3 files changed, 23 insertions(+), 10 deletions(-)
include/linux/migrate_mode.h +1 -0
@@ -23,6 +23,7 @@ enum migrate_mode {MIGRATE_MODE_MASK =3 ,MIGRATE_SINGLETHREAD =0 ,MIGRATE_MT =1 <<4 , + MIGRATE_DMA = 1<<5, };#endif /* MIGRATE_MODE_H_INCLUDED */
include/uapi/linux/mempolicy.h +1 -0
@@ -48,6 +48,7 @@ enum {#define MPOL_MF_LAZY (1<<3)/* Modifies '_MOVE: lazy migrate on fault */ #define MPOL_MF_INTERNAL (1<<4)/* Internal flags start here */ +#define MPOL_MF_MOVE_DMA (1<<5)/* Use DMA page copy routine */ #define MPOL_MF_MOVE_MT (1<<6)/* Use multi-threaded page copy routine */ #define MPOL_MF_VALID (MPOL_MF_STRICT | \
mm/migrate.c +21 -10
@@ -553,15 +553,21 @@ int migrate_huge_page_move_mapping(struct address_space *mapping,* specialized. */ static void __copy_gigantic_page (struct page *dst,struct page *src, -int nr_pages) +int nr_pages,enum migrate_mode mode) {int i;struct page *dst_base = dst;struct page *src_base = src; +int rc = -EFAULT ;for (i =0 ; i < nr_pages; ) {cond_resched (); -copy_highpage (dst, src); + +if (mode &MIGRATE_DMA ) + rc =copy_page_dma (dst, src,1 ); + +if (rc) +copy_highpage (dst, src); i++; dst =mem_map_next (dst, dst_base, i);
@@ -582,7 +588,7 @@ static void copy_huge_page(struct page *dst, struct page *src, nr_pages =pages_per_huge_page (h);if (unlikely (nr_pages >MAX_ORDER_NR_PAGES )) { -__copy_gigantic_page (dst, src, nr_pages); +__copy_gigantic_page (dst, src, nr_pages, mode);return ; } }else {
@@ -597,6 +603,8 @@ static void copy_huge_page(struct page *dst, struct page *src,if (mode &MIGRATE_MT ) rc =copy_page_multithread (dst, src, nr_pages); +else if (mode &MIGRATE_DMA ) + rc =copy_page_dma (dst, src, nr_pages);if (rc)for (i =0 ; i < nr_pages; i++) {
@@ -674,8 +682,9 @@ void migrate_page_copy(struct page *newpage, struct page *page, {if (PageHuge (page) ||PageTransHuge (page))copy_huge_page (newpage, page, mode); -else +else {copy_highpage (newpage, page); + }migrate_page_states (newpage, page); }
@@ -1511,7 +1520,8 @@ static int store_status(int __user *status, int start, int value, int nr) }static int do_move_pages_to_node (struct mm_struct *mm, -struct list_head *pagelist,int node,bool migrate_mt) +struct list_head *pagelist,int node, +bool migrate_mt,bool migrate_dma) {int err;
@@ -1519,7 +1529,8 @@ static int do_move_pages_to_node(struct mm_struct *mm,return 0 ; err =migrate_pages (pagelist, alloc_new_node_page,NULL , node, -MIGRATE_SYNC | (migrate_mt ?MIGRATE_MT :MIGRATE_SINGLETHREAD ), +MIGRATE_SYNC | (migrate_mt ?MIGRATE_MT :MIGRATE_SINGLETHREAD ) | + (migrate_dma ?MIGRATE_DMA :MIGRATE_SINGLETHREAD ),MR_SYSCALL );if (err)putback_movable_pages (pagelist);
@@ -1642,7 +1653,7 @@ static int do_pages_move(struct mm_struct *mm, nodemask_t task_nodes, start = i; }else if (node != current_node) { err =do_move_pages_to_node (mm, &pagelist, current_node, - flags &MPOL_MF_MOVE_MT ); + flags &MPOL_MF_MOVE_MT , flags &MPOL_MF_MOVE_DMA );if (err) goto out; err =store_status (status, start, current_node, i - start);
@@ -1666,7 +1677,7 @@ static int do_pages_move(struct mm_struct *mm, nodemask_t task_nodes, goto out_flush; err =do_move_pages_to_node (mm, &pagelist, current_node, - flags &MPOL_MF_MOVE_MT ); + flags &MPOL_MF_MOVE_MT , flags &MPOL_MF_MOVE_DMA );if (err) goto out;if (i > start) {
@@ -1682,7 +1693,7 @@ static int do_pages_move(struct mm_struct *mm, nodemask_t task_nodes,/* Make sure we do not overwrite the existing error */ err1 =do_move_pages_to_node (mm, &pagelist, current_node, - flags &MPOL_MF_MOVE_MT ); + flags &MPOL_MF_MOVE_MT , flags &MPOL_MF_MOVE_DMA );if (!err1) err1 =store_status (status, start, current_node, i - start);if (!err)
@@ -1778,7 +1789,7 @@ static int kernel_move_pages(pid_t pid, unsigned long nr_pages,nodemask_t task_nodes;/* Check flags */ -if (flags & ~(MPOL_MF_MOVE |MPOL_MF_MOVE_ALL |MPOL_MF_MOVE_MT )) +if (flags & ~(MPOL_MF_MOVE |MPOL_MF_MOVE_ALL |MPOL_MF_MOVE_MT |MPOL_MF_MOVE_DMA ))return -EINVAL ;if ((flags &MPOL_MF_MOVE_ALL ) && !capable (CAP_SYS_NICE ))
Signature · 1 line
2.7.4
RFC PATCH 9/25 mm: migrate: Add copy_page_lists_dma_always to support copy a list of pages.
In reply to Zi YanFrom: Zi Yan <[email protected]>
Both src and dst page lists should match the page size at each page and the length of both lists is shared.
--- mm/copy_page.c | 166 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++ mm/internal.h | 4 ++ 2 files changed, 170 insertions(+)
mm/copy_page.c +166 -0
@@ -417,3 +417,169 @@ int copy_page_dma(struct page *to, struct page *from, int nr_pages)return copy_page_dma_always (to, from, nr_pages); } + +/* +* Use DMA copy a list of pages to a new location +* +* Just put each page into individual DMA channel. +* +* */ +int copy_page_lists_dma_always (struct page **to,struct page **from,int nr_items) +{ +struct dma_async_tx_descriptor **tx =NULL ; +dma_cookie_t *cookie =NULL ; +enum dma_ctrl_flags flags[NUM_AVAIL_DMA_CHAN ] = {0 }; +struct dmaengine_unmap_data *unmap[NUM_AVAIL_DMA_CHAN ] = {0 }; +int ret_val =0 ; +int total_available_chans =NUM_AVAIL_DMA_CHAN ; +int i; +int page_idx; + +for (i =0 ; i <NUM_AVAIL_DMA_CHAN ; ++i) { +if (!copy_chan[i]) { + total_available_chans = i; + } + } +if (total_available_chans !=NUM_AVAIL_DMA_CHAN ) { +pr_err ("%d channels are missing\n" ,NUM_AVAIL_DMA_CHAN - total_available_chans); + } +if (limit_dma_chans < total_available_chans) + total_available_chans = limit_dma_chans; + +/* round down to closest 2^x value */ + total_available_chans =1 <<ilog2 (total_available_chans); + + total_available_chans =min_t (int, total_available_chans, nr_items); + + + tx =kzalloc (sizeof (struct dma_async_tx_descriptor *)*nr_items,GFP_KERNEL ); +if (!tx) { + ret_val = -ENOMEM ; + goto out; + } + cookie =kzalloc (sizeof (dma_cookie_t)*nr_items,GFP_KERNEL ); +if (!cookie) { + ret_val = -ENOMEM ; + goto out_free_tx; + } + +for (i =0 ; i < total_available_chans; ++i) { +int num_xfer_per_dev = nr_items / total_available_chans; + +if (i < (nr_items % total_available_chans)) + num_xfer_per_dev +=1 ; + +if (num_xfer_per_dev >128 ) { + ret_val = -ENOMEM ; +pr_err ("%s: too many pages to be transferred\n" , __func__); + goto out_free_both; + } + + unmap[i] =dmaengine_get_unmap_data (copy_dev[i]->dev, +2 * num_xfer_per_dev,GFP_NOWAIT ); +if (!unmap[i]) { +pr_err ("%s: no unmap data at chan %d\n" , __func__, i); + ret_val = -ENODEV ; + goto unmap_dma; + } + } + + page_idx =0 ; +for (i =0 ; i < total_available_chans; ++i) { +int num_xfer_per_dev = nr_items / total_available_chans; +int xfer_idx; + +if (i < (nr_items % total_available_chans)) + num_xfer_per_dev +=1 ; + + unmap[i]->to_cnt = num_xfer_per_dev; + unmap[i]->from_cnt = num_xfer_per_dev; + unmap[i]->len =hpage_nr_pages (from[i]) *PAGE_SIZE ; + +for (xfer_idx =0 ; xfer_idx < num_xfer_per_dev; ++xfer_idx, ++page_idx) { +size_t page_len =hpage_nr_pages (from[page_idx]) *PAGE_SIZE ; + +BUG_ON (page_len !=hpage_nr_pages (to[page_idx]) *PAGE_SIZE ); +BUG_ON (unmap[i]->len != page_len); + + unmap[i]->addr[xfer_idx] = +dma_map_page (copy_dev[i]->dev, from[page_idx], +0 , + page_len, +DMA_TO_DEVICE ); + + unmap[i]->addr[xfer_idx+num_xfer_per_dev] = +dma_map_page (copy_dev[i]->dev, to[page_idx], +0 , + page_len, +DMA_FROM_DEVICE ); + } + } + + page_idx =0 ; +for (i =0 ; i < total_available_chans; ++i) { +int num_xfer_per_dev = nr_items / total_available_chans; +int xfer_idx; + +if (i < (nr_items % total_available_chans)) + num_xfer_per_dev +=1 ; + +for (xfer_idx =0 ; xfer_idx < num_xfer_per_dev; ++xfer_idx, ++page_idx) { + + tx[page_idx] = copy_dev[i]->device_prep_dma_memcpy (copy_chan[i], + unmap[i]->addr[xfer_idx + num_xfer_per_dev], + unmap[i]->addr[xfer_idx], + unmap[i]->len, + flags[i]); +if (!tx[page_idx]) { +pr_err ("%s: no tx descriptor at chan %d xfer %d\n" , + __func__, i, xfer_idx); + ret_val = -ENODEV ; + goto unmap_dma; + } + + cookie[page_idx] = tx[page_idx]->tx_submit (tx[page_idx]); + +if (dma_submit_error (cookie[page_idx])) { +pr_err ("%s: submission error at chan %d xfer %d\n" , + __func__, i, xfer_idx); + ret_val = -ENODEV ; + goto unmap_dma; + } + } + +dma_async_issue_pending (copy_chan[i]); + } + + page_idx =0 ; +for (i =0 ; i < total_available_chans; ++i) { +int num_xfer_per_dev = nr_items / total_available_chans; +int xfer_idx; + +if (i < (nr_items % total_available_chans)) + num_xfer_per_dev +=1 ; + +for (xfer_idx =0 ; xfer_idx < num_xfer_per_dev; ++xfer_idx, ++page_idx) { + +if (dma_sync_wait (copy_chan[i], cookie[page_idx]) !=DMA_COMPLETE ) { + ret_val =-6 ; +pr_err ("%s: dma does not complete at chan %d, xfer %d\n" , + __func__, i, xfer_idx); + } + } + } + +unmap_dma: +for (i =0 ; i < total_available_chans; ++i) { +if (unmap[i]) +dmaengine_unmap_put (unmap[i]); + } + +out_free_both: +kfree (cookie); +out_free_tx: +kfree (tx); +out: + +return ret_val; +}
mm/internal.h +4 -0
@@ -555,4 +555,8 @@ static inline bool is_migrate_highatomic_page(struct page *page)void setup_zone_pageset (struct zone *zone);extern struct page *alloc_new_node_page (struct page *page,unsigned long node); + +extern int copy_page_lists_dma_always (struct page **to, +struct page **from,int nr_pages); +#endif /* __MM_INTERNAL_H */
Signature · 1 line
2.7.4
RFC PATCH 10/25 mm: migrate: copy_page_lists_mt() to copy a page list using multi-threads.
In reply to Zi YanFrom: Zi Yan <[email protected]>
This prepare the support for migrate_page_concur(), which migrates multiple pages at the same time.
--- mm/copy_page.c | 123 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++ mm/internal.h | 2 + 2 files changed, 125 insertions(+)
mm/copy_page.c +123 -0
@@ -126,6 +126,129 @@ int copy_page_multithread(struct page *to, struct page *from, int nr_pages)return err; } + +int copy_page_lists_mt (struct page **to,struct page **from,int nr_items) +{ +int err =0 ; +unsigned int total_mt_num = limit_mt_num; +int to_node =page_to_nid (*to); +int i; +struct copy_page_info *work_items[NR_CPUS ] = {0 }; +const struct cpumask *per_node_cpumask =cpumask_of_node (to_node); +int cpu_id_list[NR_CPUS ] = {0 }; +int cpu; +int max_items_per_thread; +int item_idx; + + total_mt_num = min_t(unsigned int, total_mt_num, +cpumask_weight (per_node_cpumask)); + + +if (total_mt_num >num_online_cpus ()) +return -ENODEV ; + +/* Each threads get part of each page, if nr_items < totla_mt_num */ +if (nr_items < total_mt_num) + max_items_per_thread = nr_items; +else + max_items_per_thread = (nr_items / total_mt_num) + + ((nr_items % total_mt_num)?1 :0 ); + + +for (cpu =0 ; cpu < total_mt_num; ++cpu) { + work_items[cpu] =kzalloc (sizeof (struct copy_page_info ) + +sizeof (struct copy_item )*max_items_per_thread,GFP_KERNEL ); +if (!work_items[cpu]) { + err = -ENOMEM ; + goto free_work_items; + } + } + + i =0 ; +for_each_cpu (cpu, per_node_cpumask) { +if (i >= total_mt_num) +break ; + cpu_id_list[i] = cpu; + ++i; + } + +if (nr_items < total_mt_num) { +for (cpu =0 ; cpu < total_mt_num; ++cpu) { +INIT_WORK ((struct work_struct *)work_items[cpu], + copy_page_work_queue_thread); + work_items[cpu]->num_items = max_items_per_thread; + } + +for (item_idx =0 ; item_idx < nr_items; ++item_idx) { +unsigned long chunk_size =PAGE_SIZE *hpage_nr_pages (from[item_idx]) / total_mt_num; +char *vfrom =kmap (from[item_idx]); +char *vto =kmap (to[item_idx]); +VM_BUG_ON (PAGE_SIZE *hpage_nr_pages (from[item_idx]) % total_mt_num); +BUG_ON (hpage_nr_pages (to[item_idx]) != +hpage_nr_pages (from[item_idx])); + +for (cpu =0 ; cpu < total_mt_num; ++cpu) { + work_items[cpu]->item_list[item_idx].to = vto + chunk_size * cpu; + work_items[cpu]->item_list[item_idx].from = vfrom + chunk_size * cpu; + work_items[cpu]->item_list[item_idx].chunk_size = + chunk_size; + } + } + +for (cpu =0 ; cpu < total_mt_num; ++cpu) +queue_work_on (cpu_id_list[cpu], + system_highpri_wq, + (struct work_struct *)work_items[cpu]); + }else { + item_idx =0 ; +for (cpu =0 ; cpu < total_mt_num; ++cpu) { +int num_xfer_per_thread = nr_items / total_mt_num; +int per_cpu_item_idx; + +if (cpu < (nr_items % total_mt_num)) + num_xfer_per_thread +=1 ; + +INIT_WORK ((struct work_struct *)work_items[cpu], + copy_page_work_queue_thread); + + work_items[cpu]->num_items = num_xfer_per_thread; +for (per_cpu_item_idx =0 ; per_cpu_item_idx < work_items[cpu]->num_items; + ++per_cpu_item_idx, ++item_idx) { + work_items[cpu]->item_list[per_cpu_item_idx].to =kmap (to[item_idx]); + work_items[cpu]->item_list[per_cpu_item_idx].from = +kmap (from[item_idx]); + work_items[cpu]->item_list[per_cpu_item_idx].chunk_size = +PAGE_SIZE *hpage_nr_pages (from[item_idx]); + +BUG_ON (hpage_nr_pages (to[item_idx]) != +hpage_nr_pages (from[item_idx])); + } + +queue_work_on (cpu_id_list[cpu], + system_highpri_wq, + (struct work_struct *)work_items[cpu]); + } +if (item_idx != nr_items) +pr_err ("%s: only %d out of %d pages are transferred\n" , __func__, + item_idx -1 , nr_items); + } + +/* Wait until it finishes */ +for (i =0 ; i < total_mt_num; ++i) +flush_work ((struct work_struct *)work_items[i]); + +for (i =0 ; i < nr_items; ++i) { +kunmap (to[i]); +kunmap (from[i]); + } + +free_work_items: +for (cpu =0 ; cpu < total_mt_num; ++cpu) +if (work_items[cpu]) +kfree (work_items[cpu]); + +return err; +}/* ======================== DMA copy page ======================== */ #include <linux/dmaengine.h> #include <linux/dma-mapping.h>
mm/internal.h +2 -0
@@ -558,5 +558,7 @@ extern struct page *alloc_new_node_page(struct page *page, unsigned long node);extern int copy_page_lists_dma_always (struct page **to,struct page **from,int nr_pages); +extern int copy_page_lists_mt (struct page **to, +struct page **from,int nr_pages);#endif /* __MM_INTERNAL_H */
Signature · 1 line
2.7.4
RFC PATCH 11/25 mm: migrate: Add concurrent page migration into move_pages syscall.
In reply to Zi YanFrom: Zi Yan <[email protected]>
Concurrent page migration unmaps all pages in a list, copy all pages in one function (copy_page_list*), finally remaps all new pages. This is different from existing page migration process which migrate one page at a time.
Only anonymous pages are supported. All file-backed pages are still migrated sequentially. Because locking becomes more complicated when a list of file-backed pages belong to different files, which might cause deadlocks if locks on each file are not done properly.
--- include/linux/migrate.h | 6 + include/linux/migrate_mode.h | 1 + include/uapi/linux/mempolicy.h | 1 + mm/migrate.c | 543 ++++++++++++++++++++++++++++++++++++++++- 4 files changed, 542 insertions(+), 9 deletions(-)
include/linux/migrate.h +6 -0
@@ -67,6 +67,8 @@ extern int migrate_page(struct address_space *mapping, enum migrate_mode mode);extern int migrate_pages (struct list_head *l,new_page_t new,free_page_t free,unsigned long private,enum migrate_mode mode,int reason); +extern int migrate_pages_concur (struct list_head *l,new_page_t new,free_page_t free, +unsigned long private,enum migrate_mode mode,int reason);extern int isolate_movable_page (struct page *page,isolate_mode_t mode);extern void putback_movable_page (struct page *page);
@@ -87,6 +89,10 @@ static inline int migrate_pages(struct list_head *l, new_page_t new, free_page_t free, unsigned long private, enum migrate_mode mode, int reason) { return -ENOSYS; } +static inline int migrate_pages_concur (struct list_head *l,new_page_t new, +free_page_t free,unsigned long private,enum migrate_mode mode, +int reason) + {return -ENOSYS ; }static inline int isolate_movable_page (struct page *page,isolate_mode_t mode) {return -EBUSY ; }
include/linux/migrate_mode.h +1 -0
@@ -24,6 +24,7 @@ enum migrate_mode {MIGRATE_SINGLETHREAD =0 ,MIGRATE_MT =1 <<4 ,MIGRATE_DMA =1 <<5 , + MIGRATE_CONCUR = 1<<6, };#endif /* MIGRATE_MODE_H_INCLUDED */
include/uapi/linux/mempolicy.h +1 -0
@@ -50,6 +50,7 @@ enum {#define MPOL_MF_MOVE_DMA (1<<5)/* Use DMA page copy routine */ #define MPOL_MF_MOVE_MT (1<<6)/* Use multi-threaded page copy routine */ +#define MPOL_MF_MOVE_CONCUR (1<<7)/* Move pages in a batch */ #define MPOL_MF_VALID (MPOL_MF_STRICT | \ MPOL_MF_MOVE | \
mm/migrate.c +534 -9
591 lines not on this page · raw message
Signature · 1 line
2.7.4
RFC PATCH 12/25 exchange pages: new page migration mechanism: exchange_pages()
In reply to Zi YanFrom: Zi Yan <[email protected]>
It exchanges two pages by unmapping both first, then exchanging the data of the pages using a u64 register, and finally remapping both pages.
It saves the overheads of allocating two new pages in two back-to-back migrate_pages().
--- include/linux/exchange.h | 23 ++ include/linux/ksm.h | 4 + mm/Makefile | 1 + mm/exchange.c | 597 +++++++++++++++++++++++++++++++++++++++++++++++ mm/ksm.c | 35 +++ 5 files changed, 660 insertions(+) create mode 100644 include/linux/exchange.h create mode 100644 mm/exchange.c
include/linux/exchange.h new file +23 -0
@@ -0,0 +1,23 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_EXCHANGE_H +#define _LINUX_EXCHANGE_H + +#include <linux/migrate.h> + +struct exchange_page_info { +struct page *from_page; +struct page *to_page; + +struct anon_vma *from_anon_vma; +struct anon_vma *to_anon_vma; + +int from_page_was_mapped; +int to_page_was_mapped; + +struct list_head list; +}; + +int exchange_pages (struct list_head *exchange_list, +enum migrate_mode mode, +int reason); +#endif /* _LINUX_EXCHANGE_H */
include/linux/ksm.h +4 -0
@@ -55,6 +55,7 @@ void rmap_walk_ksm(struct page *page, struct rmap_walk_control *rwc);void ksm_migrate_page (struct page *newpage,struct page *oldpage);bool reuse_ksm_page (struct page *page,struct vm_area_struct *vma,unsigned long address); +void ksm_exchange_page (struct page *to_page,struct page *from_page);#else /* !CONFIG_KSM */
@@ -92,6 +93,9 @@ static inline bool reuse_ksm_page(struct page *page, struct vm_area_struct *vma, unsigned long address) { return false; +static inline void ksm_exchange_page (struct page *to_page, +struct page *from_page) +{ }#endif /* CONFIG_MMU */ #endif /* !CONFIG_KSM */
mm/Makefile +1 -0
@@ -45,6 +45,7 @@ obj-y += init-mm.o obj-y += memblock.o obj-y += copy_page.o +obj-y += exchange.o ifdef CONFIG_MMU obj-$(CONFIG_ADVISE_SYSCALLS) += madvise.o
mm/exchange.c new file +597 -0
597 lines not on this page · raw message
mm/ksm.c +35 -0
@@ -2716,6 +2716,41 @@ void ksm_migrate_page(struct page *newpage, struct page *oldpage) set_page_stable_node(oldpage, NULL); } } + +void ksm_exchange_page (struct page *to_page,struct page *from_page) +{ +struct stable_node *to_stable_node, *from_stable_node; + +VM_BUG_ON_PAGE (!PageLocked (to_page), to_page); +VM_BUG_ON_PAGE (!PageLocked (from_page), from_page); + + to_stable_node =page_stable_node (to_page); + from_stable_node =page_stable_node (from_page); +if (to_stable_node) { +VM_BUG_ON_PAGE (to_stable_node->kpfn !=page_to_pfn (from_page), + from_page); + to_stable_node->kpfn =page_to_pfn (to_page); +/* +* newpage->mapping was set in advance; now we need smp_wmb() +* to make sure that the new stable_node->kpfn is visible +* to get_ksm_page() before it can see that oldpage->mapping +* has gone stale (or that PageSwapCache has been cleared). +*/ +smp_wmb (); + } +if (from_stable_node) { +VM_BUG_ON_PAGE (from_stable_node->kpfn !=page_to_pfn (to_page), + to_page); + from_stable_node->kpfn =page_to_pfn (from_page); +/* +* newpage->mapping was set in advance; now we need smp_wmb() +* to make sure that the new stable_node->kpfn is visible +* to get_ksm_page() before it can see that oldpage->mapping +* has gone stale (or that PageSwapCache has been cleared). +*/ +smp_wmb (); + } +}#endif /* CONFIG_MIGRATION */ #ifdef CONFIG_MEMORY_HOTREMOVE
Signature · 1 line
2.7.4
RFC PATCH 13/25 exchange pages: add multi-threaded exchange pages.
In reply to Zi YanFrom: Zi Yan <[email protected]>
Exchange two pages using multi threads. Exchange two lists of pages using multi threads.
--- mm/Makefile | 1 + mm/exchange.c | 15 ++-- mm/exchange_page.c | 229 +++++++++++++++++++++++++++++++++++++++++++++++++++++ mm/internal.h | 5 ++ 4 files changed, 245 insertions(+), 5 deletions(-) create mode 100644 mm/exchange_page.c
mm/Makefile +1 -0
@@ -46,6 +46,7 @@ obj-y += memblock.o obj-y += copy_page.o obj-y += exchange.o +obj-y += exchange_page.o ifdef CONFIG_MMU obj-$(CONFIG_ADVISE_SYSCALLS) += madvise.o
mm/exchange.c +10 -5
@@ -345,11 +345,16 @@ static int exchange_from_to_pages(struct page *to_page, struct page *from_page, rc = -EFAULT ; -if (PageHuge (from_page) ||PageTransHuge (from_page)) -exchange_huge_page (to_page, from_page); -else -exchange_highpage (to_page, from_page); - rc =0 ; +if (mode &MIGRATE_MT ) + rc =exchange_page_mthread (to_page, from_page, +hpage_nr_pages (from_page)); +if (rc) { +if (PageHuge (from_page) ||PageTransHuge (from_page)) +exchange_huge_page (to_page, from_page); +else +exchange_highpage (to_page, from_page); + rc =0 ; + }exchange_page_flags (to_page, from_page);
mm/exchange_page.c new file +229 -0
229 lines not on this page · raw message
mm/internal.h +5 -0
@@ -561,4 +561,9 @@ extern int copy_page_lists_dma_always(struct page **to,extern int copy_page_lists_mt (struct page **to,struct page **from,int nr_pages); +extern int exchange_page_mthread (struct page *to,struct page *from, +int nr_pages); +extern int exchange_page_lists_mthread (struct page **to, +struct page **from, +int nr_pages);#endif /* __MM_INTERNAL_H */
Signature · 1 line
2.7.4
RFC PATCH 14/25 exchange pages: concurrent exchange pages.
In reply to Zi YanFrom: Zi Yan <[email protected]>
It unmaps two lists of pages, then exchange them in exchange_page_lists_mthread(), and finally remaps both lists of pages.
--- include/linux/exchange.h | 2 + mm/exchange.c | 397 +++++++++++++++++++++++++++++++++++++++++++++++ mm/exchange_page.c | 1 - 3 files changed, 399 insertions(+), 1 deletion(-)
include/linux/exchange.h +2 -0
@@ -20,4 +20,6 @@ struct exchange_page_info {int exchange_pages (struct list_head *exchange_list,enum migrate_mode mode,int reason); +int exchange_pages_concur (struct list_head *exchange_list, +enum migrate_mode mode,int reason);#endif /* _LINUX_EXCHANGE_H */
mm/exchange.c +397 -0
400 lines not on this page · raw message
mm/exchange_page.c +0 -1
@@ -126,7 +126,6 @@ int exchange_page_lists_mthread(struct page **to, struct page **from, int nr_pagint to_node =page_to_nid (*to);int i;struct copy_page_info *work_items; -int nr_pages_per_page =hpage_nr_pages (*from);const struct cpumask *per_node_cpumask =cpumask_of_node (to_node);int cpu_id_list[32 ] = {0 };int cpu;
Signature · 1 line
2.7.4
RFC PATCH 15/25 exchange pages: exchange anonymous page and file-backed page.
In reply to Zi YanFrom: Zi Yan <[email protected]>
This is only done for the basic exchange pages, because we might need to lock multiple files when doing concurrent exchange pages, which could cause deadlocks easily.
--- mm/exchange.c | 284 ++++++++++++++++++++++++++++++++++++++++++++++------------ mm/internal.h | 9 ++ mm/migrate.c | 6 +- 3 files changed, 241 insertions(+), 58 deletions(-)
mm/exchange.c +229 -55
@@ -20,6 +20,8 @@#include <linux/memcontrol.h> #include <linux/balloon_compaction.h> #include <linux/buffer_head.h> +#include <linux/fs.h> /* buffer_migrate_page */ +#include <linux/backing-dev.h> #include "internal.h"
@@ -147,8 +149,6 @@ static void exchange_page_flags(struct page *to_page, struct page *from_page) from_page_flags.page_is_idle =page_is_idle (from_page);clear_page_idle (from_page); from_page_flags.page_swapcache =PageSwapCache (from_page); - from_page_flags.page_private =PagePrivate (from_page); -ClearPagePrivate (from_page); from_page_flags.page_writeback =test_clear_page_writeback (from_page);
@@ -170,8 +170,6 @@ static void exchange_page_flags(struct page *to_page, struct page *from_page) to_page_flags.page_is_idle =page_is_idle (to_page);clear_page_idle (to_page); to_page_flags.page_swapcache =PageSwapCache (to_page); - to_page_flags.page_private =PagePrivate (to_page); -ClearPagePrivate (to_page); to_page_flags.page_writeback =test_clear_page_writeback (to_page);/* set to_page */
@@ -268,18 +266,22 @@ static void exchange_page_flags(struct page *to_page, struct page *from_page)static int exchange_page_move_mapping (struct address_space *to_mapping,struct address_space *from_mapping,struct page *to_page,struct page *from_page, +struct buffer_head *to_head,struct buffer_head *from_head,enum migrate_mode mode,int to_extra_count,int from_extra_count) { -int to_expected_count =1 + to_extra_count, - from_expected_count =1 + from_extra_count; -unsigned long from_page_index =page_index (from_page), - to_page_index =page_index (to_page); +int to_expected_count =expected_page_refs (to_mapping, to_page) + to_extra_count, + from_expected_count =expected_page_refs (from_mapping, from_page) + from_extra_count; +unsigned long from_page_index = from_page->index; +unsigned long to_page_index = to_page->index;int to_swapbacked =PageSwapBacked (to_page), from_swapbacked =PageSwapBacked (from_page); -struct address_space *to_mapping_value = to_page->mapping, - *from_mapping_value = from_page->mapping; +struct address_space *to_mapping_value = to_page->mapping; +struct address_space *from_mapping_value = from_page->mapping; +VM_BUG_ON_PAGE (to_mapping !=page_mapping (to_page), to_page); +VM_BUG_ON_PAGE (from_mapping !=page_mapping (from_page), from_page); +VM_BUG_ON (PageCompound (from_page) !=PageCompound (to_page));if (!to_mapping) {/* Anonymous page without mapping */
@@ -293,26 +295,125 @@ static int exchange_page_move_mapping(struct address_space *to_mapping,return -EAGAIN ; } -/* -* Now we know that no one else is looking at the page: -* no turning back from here. -*/ -/* from_page */ - from_page->index = to_page_index; - from_page->mapping = to_mapping_value; +/* both are anonymous pages */ +if (!from_mapping && !to_mapping) { +/* from_page */ + from_page->index = to_page_index; + from_page->mapping = to_mapping_value; + +ClearPageSwapBacked (from_page); +if (to_swapbacked) +SetPageSwapBacked (from_page); + + +/* to_page */ + to_page->index = from_page_index; + to_page->mapping = from_mapping_value; + +ClearPageSwapBacked (to_page); +if (from_swapbacked) +SetPageSwapBacked (to_page); + }else if (!from_mapping && to_mapping) { +/* from is anonymous, to is file-backed */ +XA_STATE (to_xas, &to_mapping->i_pages,page_index (to_page)); +struct zone *from_zone, *to_zone; +int dirty; + + from_zone =page_zone (from_page); + to_zone =page_zone (to_page); + +xas_lock_irq (&to_xas); + +if (page_count (to_page) != to_expected_count || +xas_load (&to_xas) != to_page) { +xas_unlock_irq (&to_xas); +return -EAGAIN ; + } + +if (!page_ref_freeze (to_page, to_expected_count)) { +xas_unlock_irq (&to_xas); +pr_debug ("cannot freeze page count\n" ); +return -EAGAIN ; + } + +if (!page_ref_freeze (from_page, from_expected_count)) { +page_ref_unfreeze (to_page, to_expected_count); +xas_unlock_irq (&to_xas); + +return -EAGAIN ; + } +/* +* Now we know that no one else is looking at the page: +* no turning back from here. +*/ +ClearPageSwapBacked (from_page); +ClearPageSwapBacked (to_page); + +/* from_page */ + from_page->index = to_page_index; + from_page->mapping = to_mapping_value; +/* to_page */ + to_page->index = from_page_index; + to_page->mapping = from_mapping_value; + +if (to_swapbacked) +__SetPageSwapBacked (from_page); +else +VM_BUG_ON_PAGE (PageSwapCache (to_page), to_page); -ClearPageSwapBacked (from_page); -if (to_swapbacked) -SetPageSwapBacked (from_page); +if (from_swapbacked) +__SetPageSwapBacked (to_page); +else +VM_BUG_ON_PAGE (PageSwapCache (from_page), from_page); + dirty =PageDirty (to_page); -/* to_page */ - to_page->index = from_page_index; - to_page->mapping = from_mapping_value; +xas_store (&to_xas, from_page); +if (PageTransHuge (to_page)) { +int i; +for (i =1 ; i <HPAGE_PMD_NR ; i++) { +xas_next (&to_xas); +xas_store (&to_xas, from_page + i); + } + } + +/* move cache reference */ +page_ref_unfreeze (to_page, to_expected_count -hpage_nr_pages (to_page)); +page_ref_unfreeze (from_page, from_expected_count +hpage_nr_pages (from_page)); + +xas_unlock (&to_xas); + +/* +* If moved to a different zone then also account +* the page for that zone. Other VM counters will be +* taken care of when we establish references to the +* new page and drop references to the old page. +* +* Note that anonymous pages are accounted for +* via NR_FILE_PAGES and NR_ANON_MAPPED if they +* are mapped to swap space. +*/ +if (to_zone != from_zone) { +__dec_node_state (to_zone->zone_pgdat,NR_FILE_PAGES ); +__inc_node_state (from_zone->zone_pgdat,NR_FILE_PAGES ); +if (PageSwapBacked (to_page) && !PageSwapCache (to_page)) { +__dec_node_state (to_zone->zone_pgdat,NR_SHMEM ); +__inc_node_state (from_zone->zone_pgdat,NR_SHMEM ); + } +if (dirty &&mapping_cap_account_dirty (to_mapping)) { +__dec_node_state (to_zone->zone_pgdat,NR_FILE_DIRTY ); +__dec_zone_state (to_zone,NR_ZONE_WRITE_PENDING ); +__inc_node_state (from_zone->zone_pgdat,NR_FILE_DIRTY ); +__inc_zone_state (from_zone,NR_ZONE_WRITE_PENDING ); + } + } +local_irq_enable (); -ClearPageSwapBacked (to_page); -if (from_swapbacked) -SetPageSwapBacked (to_page); + }else { +/* from is file-backed to is anonymous: fold this to the case above */ +/* both are file-backed */ +VM_BUG_ON (1 ); + }return MIGRATEPAGE_SUCCESS ; }
@@ -322,6 +423,7 @@ static int exchange_from_to_pages(struct page *to_page, struct page *from_page, {int rc = -EBUSY ;struct address_space *to_page_mapping, *from_page_mapping; +struct buffer_head *to_head =NULL , *to_bh =NULL ;VM_BUG_ON_PAGE (!PageLocked (from_page), from_page);VM_BUG_ON_PAGE (!PageLocked (to_page), to_page);
@@ -330,15 +432,71 @@ static int exchange_from_to_pages(struct page *to_page, struct page *from_page, to_page_mapping =page_mapping (to_page); from_page_mapping =page_mapping (from_page); +/* from_page has to be anonymous page */ BUG_ON (from_page_mapping); -BUG_ON (to_page_mapping); -BUG_ON (PageWriteback (from_page)); +/* writeback has to finish */ BUG_ON (PageWriteback (to_page)); -/* actual page mapping exchange */ - rc =exchange_page_move_mapping (to_page_mapping, from_page_mapping, - to_page, from_page, mode,0 ,0 ); +/* to_page is anonymous */ +if (!to_page_mapping) { +exchange_mappings: +/* actual page mapping exchange */ + rc =exchange_page_move_mapping (to_page_mapping, from_page_mapping, + to_page, from_page,NULL ,NULL , mode,0 ,0 ); + }else { +if (to_page_mapping->a_ops->migratepage == buffer_migrate_page) { +if (!page_has_buffers (to_page)) + goto exchange_mappings; + + to_head =page_buffers (to_page); + + rc =exchange_page_move_mapping (to_page_mapping, + from_page_mapping, to_page, from_page, + to_head,NULL , mode,0 ,0 ); + +if (rc !=MIGRATEPAGE_SUCCESS ) +return rc; + +/* +* In the async case, migrate_page_move_mapping locked the buffers +* with an IRQ-safe spinlock held. In the sync case, the buffers +* need to be locked now +*/ +if ((mode &MIGRATE_MODE_MASK ) !=MIGRATE_ASYNC ) +BUG_ON (!buffer_migrate_lock_buffers (to_head, mode)); + +ClearPagePrivate (to_page); +set_page_private (from_page,page_private (to_page)); +set_page_private (to_page,0 ); +/* transfer private page count */ +put_page (to_page); +get_page (from_page); + + to_bh = to_head; +do { +set_bh_page (to_bh, from_page,bh_offset (to_bh)); + to_bh = to_bh->b_this_page; + + }while (to_bh != to_head); + +SetPagePrivate (from_page); + + to_bh = to_head; + }else if (!to_page_mapping->a_ops->migratepage) { +/* fallback_migrate_page */ +if (PageDirty (to_page)) { +if ((mode &MIGRATE_MODE_MASK ) !=MIGRATE_SYNC ) +return -EBUSY ; +return writeout (to_page_mapping, to_page); + } +if (page_has_private (to_page) && + !try_to_release_page (to_page,GFP_KERNEL )) +return -EAGAIN ; + + goto exchange_mappings; + } + }/* actual page data exchange */ if (rc !=MIGRATEPAGE_SUCCESS )return rc;
@@ -356,8 +514,28 @@ static int exchange_from_to_pages(struct page *to_page, struct page *from_page, rc =0 ; } +/* +* 1. buffer_migrate_page: +* private flag should be transferred from to_page to from_page +* +* 2. anon<->anon, fallback_migrate_page: +* both have none private flags or to_page's is cleared. +* */ +VM_BUG_ON (!((page_has_private (from_page) && !page_has_private (to_page)) || + (!page_has_private (from_page) && !page_has_private (to_page)))); +exchange_page_flags (to_page, from_page); +if (to_bh) { +VM_BUG_ON (to_bh != to_head); +do { +unlock_buffer (to_bh); +put_bh (to_bh); + to_bh = to_bh->b_this_page; + + }while (to_bh != to_head); + } +return rc; }
@@ -369,34 +547,12 @@ static int unmap_and_exchange(struct page *from_page, struct page *to_page,pgoff_t from_index, to_index;struct anon_vma *from_anon_vma =NULL , *to_anon_vma =NULL ; -/* from_page lock down */ if (!trylock_page (from_page)) {if ((mode &MIGRATE_MODE_MASK ) ==MIGRATE_ASYNC ) goto out; -lock_page (from_page); } -BUG_ON (PageWriteback (from_page)); - -/* -* By try_to_unmap(), page->mapcount goes down to 0 here. In this case, -* we cannot notice that anon_vma is freed while we migrates a page. -* This get_anon_vma() delays freeing anon_vma pointer until the end -* of migration. File cache pages are no problem because of page_lock() -* File Caches may use write_page() or lock_page() in migration, then, -* just care Anon page here. -* -* Only page_get_anon_vma() understands the subtleties of -* getting a hold on an anon_vma from outside one of its mms. -* But if we cannot get anon_vma, then we won't need it anyway, -* because that implies that the anon page is no longer mapped -* (and cannot be remapped so long as we hold the page lock). -*/ -if (PageAnon (from_page) && !PageKsm (from_page)) - from_anon_vma =page_get_anon_vma (from_page); - -/* to_page lock down */ if (!trylock_page (to_page)) {if ((mode &MIGRATE_MODE_MASK ) ==MIGRATE_ASYNC ) goto out_unlock;
@@ -404,7 +560,22 @@ static int unmap_and_exchange(struct page *from_page, struct page *to_page,lock_page (to_page); } -BUG_ON (PageWriteback (to_page)); +/* from_page is supposed to be an anonymous page */ +VM_BUG_ON_PAGE (PageWriteback (from_page), from_page); + +if (PageWriteback (to_page)) { +/* +* Only in the case of a full synchronous migration is it +* necessary to wait for PageWriteback. In the async case, +* the retry loop is too short and in the sync-light case, +* the overhead of stalling is too much +*/ +if ((mode &MIGRATE_MODE_MASK ) !=MIGRATE_SYNC ) { + rc = -EBUSY ; + goto out_unlock; + } +wait_on_page_writeback (to_page); + }/* * By try_to_unmap(), page->mapcount goes down to 0 here. In this case,
@@ -420,6 +591,9 @@ static int unmap_and_exchange(struct page *from_page, struct page *to_page,* because that implies that the anon page is no longer mapped * (and cannot be remapped so long as we hold the page lock). */ +if (PageAnon (from_page) && !PageKsm (from_page)) + from_anon_vma =page_get_anon_vma (from_page); +if (PageAnon (to_page) && !PageKsm (to_page)) to_anon_vma =page_get_anon_vma (to_page);
@@ -753,7 +927,7 @@ static int exchange_page_mapping_concur(struct list_head *unmapped_list_ptr,/* actual page mapping exchange */ rc =exchange_page_move_mapping (to_page_mapping, from_page_mapping, - to_page, from_page, mode,0 ,0 ); + to_page, from_page,NULL ,NULL , mode,0 ,0 );if (rc) {if (one_pair->from_page_was_mapped)
mm/internal.h +9 -0
@@ -566,4 +566,13 @@ extern int exchange_page_mthread(struct page *to, struct page *from,extern int exchange_page_lists_mthread (struct page **to,struct page **from,int nr_pages); + +extern int exchange_two_pages (struct page *page1,struct page *page2); + +bool buffer_migrate_lock_buffers (struct buffer_head *head, +enum migrate_mode mode); +int writeout (struct address_space *mapping,struct page *page); +int expected_page_refs (struct address_space *mapping,struct page *page); + +#endif /* __MM_INTERNAL_H */
mm/migrate.c +3 -3
@@ -385,7 +385,7 @@ void pmd_migration_entry_wait(struct mm_struct *mm, pmd_t *pmd) }#endif -static int expected_page_refs (struct address_space *mapping,struct page *page) +int expected_page_refs (struct address_space *mapping,struct page *page) {int expected_count =1 ;
@@ -732,7 +732,7 @@ EXPORT_SYMBOL(migrate_page);#ifdef CONFIG_BLOCK /* Returns true if all buffers are successfully locked */ -static bool buffer_migrate_lock_buffers (struct buffer_head *head, +bool buffer_migrate_lock_buffers (struct buffer_head *head,enum migrate_mode mode) {struct buffer_head *bh = head;
@@ -880,7 +880,7 @@ int buffer_migrate_page_norefs(struct address_space *mapping,/* * Writeback a page to clean the dirty state */ -static int writeout (struct address_space *mapping,struct page *page) +int writeout (struct address_space *mapping,struct page *page) {struct writeback_control wbc = { .sync_mode =WB_SYNC_NONE ,
Signature · 1 line
2.7.4
RFC PATCH 16/25 exchange page: Add THP exchange support.
In reply to Zi YanFrom: Zi Yan <[email protected]>
Enable exchange THPs in the process. It also need to take care of exchanging PTE-mapped THPs.
--- include/linux/exchange.h | 2 ++ mm/exchange.c | 73 +++++++++++++++++++++++++++++++++++++----------- mm/migrate.c | 2 +- 3 files changed, 60 insertions(+), 17 deletions(-)
include/linux/exchange.h +2 -0
@@ -14,6 +14,8 @@ struct exchange_page_info {int from_page_was_mapped;int to_page_was_mapped; +pgoff_t from_index, to_index; +struct list_head list; };
mm/exchange.c +57 -16
190 lines not on this page · raw message
mm/migrate.c +1 -1
@@ -229,7 +229,7 @@ static bool remove_migration_pte(struct page *page, struct vm_area_struct *vma,if (PageKsm (page)) new = page;else - new = page - pvmw.page->index + + new = page - page->index +linear_page_index (vma, pvmw.address);#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION
Signature · 1 line
2.7.4
RFC PATCH 17/25 exchange page: Add exchange_page() syscall.
In reply to Zi YanFrom: Zi Yan <[email protected]>
Users can use the syscall to exchange two lists of pages, similar to move_pages() syscall.
--- arch/x86/entry/syscalls/syscall_64.tbl | 1 + include/linux/syscalls.h | 5 + mm/exchange.c | 346 +++++++++++++++++++++++++++++++++ 3 files changed, 352 insertions(+)
arch/x86/entry/syscalls/syscall_64.tbl +1 -0
@@ -343,6 +343,7 @@ 332 common statx __x64_sys_statx 333 common io_pgetevents __x64_sys_io_pgetevents 334 common rseq __x64_sys_rseq +335 common exchange_pages __x64_sys_exchange_pages # don't use numbers 387 through 423, add new calls after the last # 'common' entry 424 common pidfd_send_signal __x64_sys_pidfd_send_signal
include/linux/syscalls.h +5 -0
@@ -1203,6 +1203,11 @@ asmlinkage long sys_mmap_pgoff(unsigned long addr, unsigned long len, unsigned long fd, unsigned long pgoff);asmlinkage long sys_old_mmap (struct mmap_arg_struct __user *arg); +asmlinkage long sys_exchange_pages (pid_t pid,unsigned long nr_pages, +const void __user *__user *from_pages, +const void __user *__user *to_pages, +int __user *status, +int flags);/* * Not a real system call, but a placeholder for syscalls which are
mm/exchange.c +346 -0
355 lines not on this page · raw message
Signature · 1 line
2.7.4
RFC PATCH 18/25 memcg: Add per node memory usage&max stats in memcg.
In reply to Zi YanFrom: Zi Yan <[email protected]>
It prepares for the following patches to enable memcg-based NUMA node page migration. We are going to limit memory usage in each node on a per-memcg basis.
--- include/linux/cgroup-defs.h | 1 + include/linux/memcontrol.h | 67 +++++++++++++++++++++++++++++++++++++ mm/memcontrol.c | 80 +++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 148 insertions(+)
include/linux/cgroup-defs.h +1 -0
@@ -531,6 +531,7 @@ struct cftype {struct cgroup_subsys *ss;/* NULL for cgroup core files */ struct list_head node;/* anchored at ss->cfts */ struct kernfs_ops *kf_ops; +int numa_node_id;int (*open)(struct kernfs_open_file *of);void (*release)(struct kernfs_open_file *of);
include/linux/memcontrol.h +67 -0
85 lines not on this page · raw message
mm/memcontrol.c +80 -0
90 lines not on this page · raw message
Signature · 1 line
2.7.4
RFC PATCH 19/25 mempolicy: add MPOL_F_MEMCG flag, enforcing memcg memory limit.
In reply to Zi YanFrom: Zi Yan <[email protected]>
With MPOL_F_MEMCG set and MPOL_PREFERRED is used, we will enforce the memory limit set in the corresponding memcg.
--- include/uapi/linux/mempolicy.h | 3 ++- mm/mempolicy.c | 36 ++++++++++++++++++++++++++++++++++++ 2 files changed, 38 insertions(+), 1 deletion(-)
include/uapi/linux/mempolicy.h +2 -1
@@ -28,12 +28,13 @@ enum {/* Flags for set_mempolicy */ #define MPOL_F_STATIC_NODES (1 << 15)#define MPOL_F_RELATIVE_NODES (1 << 14) +#define MPOL_F_MEMCG (1 << 13)/* * MPOL_MODE_FLAGS is the union of all possible optional mode flags passed to * either set_mempolicy() or mbind(). */ -#define MPOL_MODE_FLAGS (MPOL_F_STATIC_NODES | MPOL_F_RELATIVE_NODES) +#define MPOL_MODE_FLAGS (MPOL_F_STATIC_NODES | MPOL_F_RELATIVE_NODES | MPOL_F_MEMCG)/* Flags for get_mempolicy */ #define MPOL_F_NODE (1<<0)/* return next IL mode instead of node mask */
mm/mempolicy.c +36 -0
@@ -2040,6 +2040,42 @@ alloc_pages_vma(gfp_t gfp, int order, struct vm_area_struct *vma, goto out; } +if (pol->mode ==MPOL_PREFERRED && (pol->flags &MPOL_F_MEMCG )) { +struct task_struct *p = current; +struct mem_cgroup *memcg =mem_cgroup_from_task (p); +int nid = pol->v.preferred_node; +unsigned long nr_memcg_node_size; +struct mm_struct *mm =get_task_mm (p); +unsigned long nr_pages = hugepage?HPAGE_PMD_NR :1 ; + +if (!(memcg && mm)) { +if (mm) +mmput (mm); + goto use_other_policy; + } + +/* skip preferred node if mm_manage is going on */ +if (test_bit (MMF_MM_MANAGE , &mm->flags)) { + nid =next_memory_node (nid); +if (nid ==MAX_NUMNODES ) + nid = first_memory_node; + } +mmput (mm); + + nr_memcg_node_size =memcg_max_size_node (memcg, nid); + +while (nr_memcg_node_size !=ULONG_MAX && + nr_memcg_node_size <= (memcg_size_node (memcg, nid) + nr_pages)) { +if ((nid =next_memory_node (nid)) ==MAX_NUMNODES ) + nid = first_memory_node; + nr_memcg_node_size =memcg_max_size_node (memcg, nid); + } + +mpol_cond_put (pol); + page =__alloc_pages_node (nid, gfp | __GFP_THISNODE, order); + goto out; + } +use_other_policy:if (unlikely (IS_ENABLED (CONFIG_TRANSPARENT_HUGEPAGE ) && hugepage)) {int hpage_node = node;
Signature · 1 line
2.7.4
RFC PATCH 20/25 memory manage: Add memory manage syscall.
In reply to Zi YanFrom: Zi Yan <[email protected]>
This prepares for the following patches to provide a user API to manipulate pages in two memory nodes with the help of memcg.
missing memcg_max_size_node()
--- arch/x86/entry/syscalls/syscall_64.tbl | 1 + include/linux/sched/coredump.h | 1 + include/linux/syscalls.h | 5 ++ include/uapi/linux/mempolicy.h | 1 + mm/Makefile | 1 + mm/internal.h | 2 + mm/memory_manage.c | 109 +++++++++++++++++++++++++++++++++ mm/mempolicy.c | 2 +- 8 files changed, 121 insertions(+), 1 deletion(-) create mode 100644 mm/memory_manage.c
arch/x86/entry/syscalls/syscall_64.tbl +1 -0
@@ -344,6 +344,7 @@ 333 common io_pgetevents __x64_sys_io_pgetevents 334 common rseq __x64_sys_rseq 335 common exchange_pages __x64_sys_exchange_pages +336 common mm_manage __x64_sys_mm_manage # don't use numbers 387 through 423, add new calls after the last # 'common' entry 424 common pidfd_send_signal __x64_sys_pidfd_send_signal
include/linux/sched/coredump.h +1 -0
@@ -73,6 +73,7 @@ static inline int get_dumpable(struct mm_struct *mm)#define MMF_OOM_VICTIM 25/* mm is the oom victim */ #define MMF_OOM_REAP_QUEUED 26/* mm was queued for oom_reaper */ #define MMF_DISABLE_THP_MASK (1 << MMF_DISABLE_THP) +#define MMF_MM_MANAGE 27#define MMF_INIT_MASK (MMF_DUMPABLE_MASK | MMF_DUMP_FILTER_MASK |\ MMF_DISABLE_THP_MASK)
include/linux/syscalls.h +5 -0
@@ -1208,6 +1208,11 @@ asmlinkage long sys_exchange_pages(pid_t pid, unsigned long nr_pages,const void __user *__user *to_pages, int __user *status, int flags); +asmlinkage long sys_mm_manage (pid_t pid,unsigned long nr_pages, +unsigned long maxnode, +const unsigned long __user *old_nodes, +const unsigned long __user *new_nodes, +int flags);/* * Not a real system call, but a placeholder for syscalls which are
include/uapi/linux/mempolicy.h +1 -0
@@ -52,6 +52,7 @@ enum {#define MPOL_MF_MOVE_DMA (1<<5)/* Use DMA page copy routine */ #define MPOL_MF_MOVE_MT (1<<6)/* Use multi-threaded page copy routine */ #define MPOL_MF_MOVE_CONCUR (1<<7)/* Move pages in a batch */ +#define MPOL_MF_EXCHANGE (1<<8)/* Exchange pages */ #define MPOL_MF_VALID (MPOL_MF_STRICT | \ MPOL_MF_MOVE | \
mm/Makefile +1 -0
@@ -47,6 +47,7 @@ obj-y += memblock.o obj-y += copy_page.o obj-y += exchange.o obj-y += exchange_page.o +obj-y += memory_manage.o ifdef CONFIG_MMU obj-$(CONFIG_ADVISE_SYSCALLS) += madvise.o
mm/internal.h +2 -0
@@ -574,5 +574,7 @@ bool buffer_migrate_lock_buffers(struct buffer_head *head,int writeout (struct address_space *mapping,struct page *page);int expected_page_refs (struct address_space *mapping,struct page *page); +int get_nodes (nodemask_t *nodes,const unsigned long __user *nmask, +unsigned long maxnode);#endif /* __MM_INTERNAL_H */
mm/memory_manage.c new file +109 -0
110 lines not on this page · raw message
mm/mempolicy.c +1 -1
@@ -1249,7 +1249,7 @@ static long do_mbind(unsigned long start, unsigned long len,*/ /* Copy a node mask from user space. */ -static int get_nodes (nodemask_t *nodes,const unsigned long __user *nmask, +int get_nodes (nodemask_t *nodes,const unsigned long __user *nmask,unsigned long maxnode) {unsigned long k;
Signature · 1 line
2.7.4
RFC PATCH 21/25 mm: move update_lru_sizes() to mm_inline.h for broader use.
In reply to Zi YanFrom: Zi Yan <[email protected]>
--- include/linux/mm_inline.h | 21 +++++++++++++++++++++ mm/vmscan.c | 25 ++----------------------- 2 files changed, 23 insertions(+), 23 deletions(-)
include/linux/mm_inline.h +21 -0
@@ -44,6 +44,27 @@ static __always_inline void update_lru_size(struct lruvec *lruvec,#endif } +/* +* Update LRU sizes after isolating pages. The LRU size updates must +* be complete before mem_cgroup_update_lru_size due to a santity check. +*/ +static __always_inline void update_lru_sizes(struct lruvec *lruvec, +enum lru_list lru,unsigned long *nr_zone_taken) +{ +int zid; + +for (zid =0 ; zid <MAX_NR_ZONES ; zid++) { +if (!nr_zone_taken[zid]) +continue ; + +__update_lru_size (lruvec, lru, zid, -nr_zone_taken[zid]); +#ifdef CONFIG_MEMCG +mem_cgroup_update_lru_size (lruvec, lru, zid, -nr_zone_taken[zid]); +#endif + } + +} + static __always_inline void add_page_to_lru_list(struct page *page,struct lruvec *lruvec,enum lru_list lru) {
mm/vmscan.c +2 -23
@@ -1593,27 +1593,6 @@ int __isolate_lru_page(struct page *page, isolate_mode_t mode) } -/* -* Update LRU sizes after isolating pages. The LRU size updates must -* be complete before mem_cgroup_update_lru_size due to a santity check. -*/ -static __always_inline void update_lru_sizes(struct lruvec *lruvec, -enum lru_list lru,unsigned long *nr_zone_taken) -{ -int zid; - -for (zid =0 ; zid <MAX_NR_ZONES ; zid++) { -if (!nr_zone_taken[zid]) -continue ; - -__update_lru_size (lruvec, lru, zid, -nr_zone_taken[zid]); -#ifdef CONFIG_MEMCG -mem_cgroup_update_lru_size (lruvec, lru, zid, -nr_zone_taken[zid]); -#endif - } - -} -/** * pgdat->lru_lock is heavily contended. Some of the functions that * shrink the lists perform better by taking out a batch of pages
@@ -1804,7 +1783,7 @@ static int too_many_isolated(struct pglist_data *pgdat, int file,return isolated > inactive; } -static noinline_for_stack void +noinline_for_stack voidputback_inactive_pages (struct lruvec *lruvec,struct list_head *page_list) {struct zone_reclaim_stat *reclaim_stat = &lruvec->reclaim_stat;
@@ -2003,7 +1982,7 @@ shrink_inactive_list(unsigned long nr_to_scan, struct lruvec *lruvec,* Returns the number of pages moved to the given lru. */ -static unsigned move_active_pages_to_lru (struct lruvec *lruvec, +unsigned move_active_pages_to_lru (struct lruvec *lruvec,struct list_head *list,struct list_head *pages_to_free,enum lru_list lru)
Signature · 1 line
2.7.4
RFC PATCH 22/25 memory manage: active/inactive page list manipulation in memcg.
In reply to Zi YanFrom: Zi Yan <[email protected]>
The syscall allows users to trigger page list scanning to actively move pages between active/inactive lists according to page references. This is limited to the memcg which the process belongs to. It would not impact the global LRU lists, which is the root memcg.
--- include/uapi/linux/mempolicy.h | 1 + mm/internal.h | 93 +++++++++++++++++++++++++++++++++++++++++- mm/memory_manage.c | 76 +++++++++++++++++++++++++++++++++- mm/vmscan.c | 90 ++++++++-------------------------------- 4 files changed, 184 insertions(+), 76 deletions(-)
include/uapi/linux/mempolicy.h +1 -0
@@ -53,6 +53,7 @@ enum {#define MPOL_MF_MOVE_MT (1<<6)/* Use multi-threaded page copy routine */ #define MPOL_MF_MOVE_CONCUR (1<<7)/* Move pages in a batch */ #define MPOL_MF_EXCHANGE (1<<8)/* Exchange pages */ +#define MPOL_MF_SHRINK_LISTS (1<<9)/* Exchange pages */ #define MPOL_MF_VALID (MPOL_MF_STRICT | \ MPOL_MF_MOVE | \
mm/internal.h +92 -1
103 lines not on this page · raw message
mm/memory_manage.c +74 -2
103 lines not on this page · raw message
mm/vmscan.c +17 -73
132 lines not on this page · raw message
Signature · 1 line
2.7.4
RFC PATCH 23/25 memory manage: page migration based page manipulation between NUMA nodes.
In reply to Zi YanFrom: Zi Yan <[email protected]>
Users are expected to set memcg max size to reflect their memory resource allocation policy. The syscall simply migrates pages belong to the application's memcg between from_node to to_node, where from_node is considered fast memory and to_node is considered slow memory. In common cases, active(hot) pages are migrated from to_node to from_node and inactive(cold) pages are migrated from from_node to to_node.
Separate migration for base pages and huge pages to achieve high throughput.
1. They are migrated via different calls.
2. 4KB base pages are not transferred via multi-threaded.
3. All pages are migrated together if no optimization is used.
--- mm/memory_manage.c | 275 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 275 insertions(+)
mm/memory_manage.c +275 -0
@@ -6,6 +6,7 @@#include <linux/cpuset.h> #include <linux/mempolicy.h> #include <linux/memcontrol.h> +#include <linux/migrate.h> #include <linux/mm_inline.h> #include <linux/nodemask.h> #include <linux/rmap.h>
@@ -15,6 +16,11 @@#include "internal.h" +enum isolate_action { +ISOLATE_COLD_PAGES =1 , +ISOLATE_HOT_PAGES , +ISOLATE_HOT_AND_COLD_PAGES , +};static unsigned long shrink_lists_node_memcg (pg_data_t *pgdat,struct mem_cgroup *memcg,unsigned long nr_to_scan)
@@ -78,6 +84,272 @@ static int shrink_lists(struct task_struct *p, struct mm_struct *mm,return err; } +static unsigned long isolate_pages_from_lru_list (pg_data_t *pgdat, +struct mem_cgroup *memcg,unsigned long nr_pages, +struct list_head *base_page_list, +struct list_head *huge_page_list, +unsigned long *nr_taken_base_page, +unsigned long *nr_taken_huge_page, +enum isolate_action action) +{ +struct lruvec *lruvec =mem_cgroup_lruvec (pgdat, memcg); +enum lru_list lru; +unsigned long nr_all_taken =0 ; + +if (nr_pages ==ULONG_MAX ) + nr_pages =memcg_size_node (memcg, pgdat->node_id); + +lru_add_drain_all (); + +for_each_evictable_lru (lru) { +unsigned long nr_scanned, nr_taken; +int file =is_file_lru (lru); +struct scan_control sc = {.may_unmap =1 }; + +if (action ==ISOLATE_COLD_PAGES &&is_active_lru (lru)) +continue ; +if (action ==ISOLATE_HOT_PAGES && !is_active_lru (lru)) +continue ; + +spin_lock_irq (&pgdat->lru_lock); + +/* Isolate base pages */ + sc.isolate_only_base_page =1 ; + nr_taken =isolate_lru_pages (nr_pages, lruvec, base_page_list, + &nr_scanned, &sc, lru); +/* Isolate huge pages */ + sc.isolate_only_base_page =0 ; + sc.isolate_only_huge_page =1 ; + nr_taken +=isolate_lru_pages (nr_pages - nr_scanned, lruvec, + huge_page_list, &nr_scanned, &sc, lru); + +__mod_node_page_state (pgdat,NR_ISOLATED_ANON + file, nr_taken); + +spin_unlock_irq (&pgdat->lru_lock); + + nr_all_taken += nr_taken; + +if (nr_all_taken > nr_pages) +break ; + } + +return nr_all_taken; +} + +static int migrate_to_node (struct list_head *page_list,int nid, +enum migrate_mode mode) +{ +bool migrate_concur = mode &MIGRATE_CONCUR ; +int num =0 ; +int from_nid; +int err; + +if (list_empty (page_list)) +return num; + + from_nid = page_to_nid(list_first_entry(page_list, struct page, lru)); + +if (migrate_concur) + err =migrate_pages_concur (page_list, alloc_new_node_page, +NULL , nid, mode,MR_SYSCALL ); +else + err =migrate_pages (page_list, alloc_new_node_page, +NULL , nid, mode,MR_SYSCALL ); + +if (err) { +struct page *page; + +list_for_each_entry (page, page_list, lru) + num +=hpage_nr_pages (page); +pr_debug ("%d pages failed to migrate from %d to %d\n" , + num, from_nid, nid); + +putback_movable_pages (page_list); + } +return num; +} + +static inline int _putback_overflow_pages (unsigned long max_nr_pages, +struct list_head *page_list,unsigned long *nr_remaining_pages) +{ +struct page *page; +LIST_HEAD (putback_list); + +if (list_empty (page_list)) +return max_nr_pages; + + *nr_remaining_pages =0 ; +/* in case we need to drop the whole list */ + page = list_first_entry(page_list, struct page, lru); +if (max_nr_pages <= (2 *hpage_nr_pages (page))) { + max_nr_pages =0 ; +putback_movable_pages (page_list); + goto out; + } + +list_for_each_entry (page, page_list, lru) { +int nr_pages =hpage_nr_pages (page); +/* drop just one more page to avoid using up free space */ +if (max_nr_pages <= (2 * nr_pages)) { + max_nr_pages =0 ; +break ; + } + max_nr_pages -= nr_pages; + *nr_remaining_pages += nr_pages; + } + +/* we did not scan all pages in page_list, we need to put back some */ +if (&page->lru != page_list) { +list_cut_position (&putback_list, page_list, &page->lru); +putback_movable_pages (page_list); +list_splice (&putback_list, page_list); + } +out: +return max_nr_pages; +} + +static int putback_overflow_pages (unsigned long max_nr_base_pages, +unsigned long max_nr_huge_pages, +long nr_free_pages, +struct list_head *base_page_list, +struct list_head *huge_page_list, +unsigned long *nr_base_pages, +unsigned long *nr_huge_pages) +{ +if (nr_free_pages <0 ) { +if ((-nr_free_pages) > max_nr_base_pages) { + nr_free_pages += max_nr_base_pages; + max_nr_base_pages =0 ; + } + +if ((-nr_free_pages) > max_nr_huge_pages) { + nr_free_pages =0 ; + max_nr_base_pages =0 ; + } + } +/* +* counting pages in page lists and substract the number from max_nr_* +* when max_nr_* go to zero, drop the remaining pages +*/ + max_nr_huge_pages +=_putback_overflow_pages (nr_free_pages/2 + max_nr_base_pages, + base_page_list, nr_base_pages); +return _putback_overflow_pages (nr_free_pages/2 + max_nr_huge_pages, + huge_page_list, nr_huge_pages); +} + +static int do_mm_manage (struct task_struct *p,struct mm_struct *mm, +const nodemask_t *slow,const nodemask_t *fast, +unsigned long nr_pages,int flags) +{ +bool migrate_mt = flags &MPOL_MF_MOVE_MT ; +bool migrate_concur = flags &MPOL_MF_MOVE_CONCUR ; +bool migrate_dma = flags &MPOL_MF_MOVE_DMA ; +bool move_hot_and_cold_pages = flags &MPOL_MF_MOVE_ALL ; +struct mem_cgroup *memcg =mem_cgroup_from_task (p); +int err =0 ; +unsigned long nr_isolated_slow_pages; +unsigned long nr_isolated_slow_base_pages =0 ; +unsigned long nr_isolated_slow_huge_pages =0 ; +unsigned long nr_isolated_fast_pages; +/* in case no migration from to node, we migrate all isolated pages from +* slow node */ +unsigned long nr_isolated_fast_base_pages =ULONG_MAX ; +unsigned long nr_isolated_fast_huge_pages =ULONG_MAX ; +unsigned long max_nr_pages_fast_node, nr_pages_fast_node; +unsigned long nr_pages_slow_node, nr_active_pages_slow_node; +long nr_free_pages_fast_node; +int slow_nid, fast_nid; +enum migrate_mode mode =MIGRATE_SYNC | + (migrate_mt ?MIGRATE_MT :MIGRATE_SINGLETHREAD ) | + (migrate_dma ?MIGRATE_DMA :MIGRATE_SINGLETHREAD ) | + (migrate_concur ?MIGRATE_CONCUR :MIGRATE_SINGLETHREAD ); +enum isolate_action isolate_action = + move_hot_and_cold_pages?ISOLATE_HOT_AND_COLD_PAGES :ISOLATE_HOT_PAGES ; +LIST_HEAD (slow_base_page_list); +LIST_HEAD (slow_huge_page_list); + +if (!memcg) +return 0 ; +/* Let's handle simplest situation first */ +if (!(nodes_weight (*slow) ==1 &&nodes_weight (*fast) ==1 )) +return 0 ; + +/* Only work on specific cgroup not the global root */ +if (memcg == root_mem_cgroup) +return 0 ; + + slow_nid =first_node (*slow); + fast_nid =first_node (*fast); + + max_nr_pages_fast_node =memcg_max_size_node (memcg, fast_nid); + nr_pages_fast_node =memcg_size_node (memcg, fast_nid); + nr_active_pages_slow_node =active_inactive_size_memcg_node (memcg, + slow_nid, true); + nr_pages_slow_node =memcg_size_node (memcg, slow_nid); + + nr_free_pages_fast_node = max_nr_pages_fast_node - nr_pages_fast_node; + +/* do not migrate in more pages than fast node can hold */ + nr_pages = min_t(unsigned long, max_nr_pages_fast_node, nr_pages); +/* do not migrate away more pages than slow node has */ + nr_pages = min_t(unsigned long, nr_pages_slow_node, nr_pages); + +/* if fast node has enough space, migrate all possible pages in slow node */ +if (nr_pages !=ULONG_MAX && + nr_free_pages_fast_node >0 && + nr_active_pages_slow_node < nr_free_pages_fast_node) { + isolate_action =ISOLATE_HOT_AND_COLD_PAGES ; + } + + nr_isolated_slow_pages =isolate_pages_from_lru_list (NODE_DATA (slow_nid), + memcg, nr_pages, &slow_base_page_list, &slow_huge_page_list, + &nr_isolated_slow_base_pages, &nr_isolated_slow_huge_pages, + isolate_action); + +if (max_nr_pages_fast_node !=ULONG_MAX && + (nr_free_pages_fast_node <0 || + nr_free_pages_fast_node < nr_isolated_slow_pages)) { +LIST_HEAD (fast_base_page_list); +LIST_HEAD (fast_huge_page_list); + + nr_isolated_fast_base_pages =0 ; + nr_isolated_fast_huge_pages =0 ; +/* isolate pages on fast node to make space */ + nr_isolated_fast_pages =isolate_pages_from_lru_list (NODE_DATA (fast_nid), + memcg, + nr_isolated_slow_pages - nr_free_pages_fast_node, + &fast_base_page_list, &fast_huge_page_list, + &nr_isolated_fast_base_pages, &nr_isolated_fast_huge_pages, + move_hot_and_cold_pages?ISOLATE_HOT_AND_COLD_PAGES :ISOLATE_COLD_PAGES ); + +/* Migrate pages to slow node */ +/* No multi-threaded migration for base pages */ + nr_isolated_fast_base_pages -= +migrate_to_node (&fast_base_page_list, slow_nid, mode & ~MIGRATE_MT ); + + nr_isolated_fast_huge_pages -= +migrate_to_node (&fast_huge_page_list, slow_nid, mode); + } + +if (nr_isolated_fast_base_pages !=ULONG_MAX && + nr_isolated_fast_huge_pages !=ULONG_MAX ) +putback_overflow_pages (nr_isolated_fast_base_pages, + nr_isolated_fast_huge_pages, nr_free_pages_fast_node, + &slow_base_page_list, &slow_huge_page_list, + &nr_isolated_slow_base_pages, + &nr_isolated_slow_huge_pages); + +/* Migrate pages to fast node */ +/* No multi-threaded migration for base pages */ + nr_isolated_slow_base_pages -= +migrate_to_node (&slow_base_page_list, fast_nid, mode & ~MIGRATE_MT ); + + nr_isolated_slow_huge_pages -= +migrate_to_node (&slow_huge_page_list, fast_nid, mode); + +return err; +} + SYSCALL_DEFINE6(mm_manage, pid_t, pid, unsigned long, nr_pages, unsigned long, maxnode, const unsigned long __user *, slow_nodes,
@@ -167,6 +439,9 @@ SYSCALL_DEFINE6(mm_manage, pid_t, pid, unsigned long, nr_pages,if (flags &MPOL_MF_SHRINK_LISTS )shrink_lists (task, mm, slow, fast, nr_pages); +if (flags &MPOL_MF_MOVE ) + err =do_mm_manage (task, mm, slow, fast, nr_pages, flags); +clear_bit (MMF_MM_MANAGE , &mm->flags);mmput (mm); out:
Signature · 1 line
2.7.4
RFC PATCH 24/25 memory manage: limit migration batch size.
In reply to Zi YanFrom: Zi Yan <[email protected]>
Make migration batch size adjustable to avoid excessive migration overheads when a lot of pages are under migration.
--- kernel/sysctl.c | 8 ++++++++ mm/memory_manage.c | 60 ++++++++++++++++++++++++++++++++++++------------------ 2 files changed, 48 insertions(+), 20 deletions(-)
kernel/sysctl.c +8 -0
@@ -105,6 +105,7 @@ extern int accel_page_copy;extern unsigned int limit_mt_num;extern int use_all_dma_chans;extern int limit_dma_chans; +extern int migration_batch_size;/* External variables not in a header file. */ extern int suid_dumpable;
@@ -1470,6 +1471,13 @@ static struct ctl_table vm_table[] = { .extra1 = &zero, }, { + .procname = "migration_batch_size", + .data = &migration_batch_size, + .maxlen = sizeof(migration_batch_size), + .mode = 0644, + .proc_handler = proc_dointvec, + }, + { .procname = "hugetlb_shm_group", .data = &sysctl_hugetlb_shm_group, .maxlen = sizeof(gid_t),
mm/memory_manage.c +40 -20
101 lines not on this page · raw message
Signature · 1 line
2.7.4
RFC PATCH 25/25 memory manage: use exchange pages to memory manage to improve throughput.
In reply to Zi YanFrom: Zi Yan <[email protected]>
1. Exclude file-backed base pages from exchanging.
2. Split THP in exchange pages if THP support is disabled.
3. if THP migration is supported, only exchange THPs.
--- mm/memory_manage.c | 173 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 173 insertions(+)
mm/memory_manage.c +173 -0
@@ -7,6 +7,7 @@#include <linux/mempolicy.h> #include <linux/memcontrol.h> #include <linux/migrate.h> +#include <linux/exchange.h> #include <linux/mm_inline.h> #include <linux/nodemask.h> #include <linux/rmap.h>
@@ -253,6 +254,147 @@ static int putback_overflow_pages(unsigned long max_nr_base_pages, huge_page_list, nr_huge_pages); } +static int add_pages_to_exchange_list (struct list_head *from_pagelist, +struct list_head *to_pagelist,struct exchange_page_info *info_list, +struct list_head *exchange_list,unsigned long info_list_size) +{ +unsigned long info_list_index =0 ; +LIST_HEAD (failed_from_list); +LIST_HEAD (failed_to_list); + +while (!list_empty (from_pagelist) && !list_empty (to_pagelist)) { +struct page *from_page, *to_page; +struct exchange_page_info *one_pair = &info_list[info_list_index]; +int rc; + + from_page = list_first_entry_or_null(from_pagelist, struct page, lru); + to_page = list_first_entry_or_null(to_pagelist, struct page, lru); + +if (!from_page || !to_page) +break ; + +if (!thp_migration_supported () &&PageTransHuge (from_page)) { +lock_page (from_page); + rc =split_huge_page_to_list (from_page, &from_page->lru); +unlock_page (from_page); +if (rc) { +list_move (&from_page->lru, &failed_from_list); +continue ; + } + } + +if (!thp_migration_supported () &&PageTransHuge (to_page)) { +lock_page (to_page); + rc =split_huge_page_to_list (to_page, &to_page->lru); +unlock_page (to_page); +if (rc) { +list_move (&to_page->lru, &failed_to_list); +continue ; + } + } + +if (hpage_nr_pages (from_page) !=hpage_nr_pages (to_page)) { +if (!(hpage_nr_pages (from_page) ==1 &&hpage_nr_pages (from_page) ==HPAGE_PMD_NR )) { +list_del (&from_page->lru); +list_add (&from_page->lru, &failed_from_list); + } +if (!(hpage_nr_pages (to_page) ==1 &&hpage_nr_pages (to_page) ==HPAGE_PMD_NR )) { +list_del (&to_page->lru); +list_add (&to_page->lru, &failed_to_list); + } +continue ; + } + +/* Exclude file-backed pages, exchange it concurrently is not +* implemented yet. */ +if (page_mapping (from_page)) { +list_del (&from_page->lru); +list_add (&from_page->lru, &failed_from_list); +continue ; + } +if (page_mapping (to_page)) { +list_del (&to_page->lru); +list_add (&to_page->lru, &failed_to_list); +continue ; + } + +list_del (&from_page->lru); +list_del (&to_page->lru); + + one_pair->from_page = from_page; + one_pair->to_page = to_page; + +list_add_tail (&one_pair->list, exchange_list); + + info_list_index++; +if (info_list_index >= info_list_size) +break ; + } +list_splice (&failed_from_list, from_pagelist); +list_splice (&failed_to_list, to_pagelist); + +return info_list_index; +} + +static unsigned long exchange_pages_between_nodes (unsigned long nr_from_pages, +unsigned long nr_to_pages,struct list_head *from_page_list, +struct list_head *to_page_list,int batch_size, +bool huge_page,enum migrate_mode mode) +{ +struct exchange_page_info *info_list; + unsigned long info_list_size = min_t(unsigned long, + nr_from_pages, nr_to_pages) / (huge_page?HPAGE_PMD_NR :1 ); +unsigned long added_size =0 ; +bool migrate_concur = mode &MIGRATE_CONCUR ; +LIST_HEAD (exchange_list); + +/* non concurrent does not need to split into batches */ +if (!migrate_concur || batch_size <=0 ) + batch_size = info_list_size; + +/* prepare for huge page split */ +if (!thp_migration_supported () && huge_page) { + batch_size = batch_size *HPAGE_PMD_NR ; + info_list_size = info_list_size *HPAGE_PMD_NR ; + } + + info_list =kvzalloc (sizeof (struct exchange_page_info )*batch_size, +GFP_KERNEL ); +if (!info_list) +return 0 ; + +while (!list_empty (from_page_list) && !list_empty (to_page_list)) { +unsigned long nr_added_pages; +INIT_LIST_HEAD (&exchange_list); + + nr_added_pages =add_pages_to_exchange_list (from_page_list, to_page_list, + info_list, &exchange_list, batch_size); + +/* +* Nothing to exchange, we bail out. +* +* In case from_page_list and to_page_list both only have file-backed +* pages left */ +if (!nr_added_pages) +break ; + + added_size += nr_added_pages; + +VM_BUG_ON (added_size > info_list_size); + +if (migrate_concur) +exchange_pages_concur (&exchange_list, mode,MR_SYSCALL ); +else +exchange_pages (&exchange_list, mode,MR_SYSCALL ); + +memset (info_list,0 ,sizeof (struct exchange_page_info )*batch_size); + } + +kvfree (info_list); + +return info_list_size; +} +static int do_mm_manage (struct task_struct *p,struct mm_struct *mm,const nodemask_t *slow,const nodemask_t *fast,unsigned long nr_pages,int flags)
@@ -261,6 +403,7 @@ static int do_mm_manage(struct task_struct *p, struct mm_struct *mm,bool migrate_concur = flags &MPOL_MF_MOVE_CONCUR ;bool migrate_dma = flags &MPOL_MF_MOVE_DMA ;bool move_hot_and_cold_pages = flags &MPOL_MF_MOVE_ALL ; +bool migrate_exchange_pages = flags &MPOL_MF_EXCHANGE ;struct mem_cgroup *memcg =mem_cgroup_from_task (p);int err =0 ;unsigned long nr_isolated_slow_pages;
@@ -338,6 +481,35 @@ static int do_mm_manage(struct task_struct *p, struct mm_struct *mm, &nr_isolated_fast_base_pages, &nr_isolated_fast_huge_pages, move_hot_and_cold_pages?ISOLATE_HOT_AND_COLD_PAGES:ISOLATE_COLD_PAGES); +if (migrate_exchange_pages) { +unsigned long nr_exchange_pages; + +/* +* base pages can include file-backed ones, we do not handle them +* at the moment +*/ +if (!thp_migration_supported ()) { + nr_exchange_pages =exchange_pages_between_nodes (nr_isolated_slow_base_pages, + nr_isolated_fast_base_pages, &slow_base_page_list, + &fast_base_page_list, migration_batch_size, false, mode); + + nr_isolated_fast_base_pages -= nr_exchange_pages; + } + +/* THP page exchange */ + nr_exchange_pages =exchange_pages_between_nodes (nr_isolated_slow_huge_pages, + nr_isolated_fast_huge_pages, &slow_huge_page_list, + &fast_huge_page_list, migration_batch_size, true, mode); + +/* split THP above, so we do not need to multiply the counter */ +if (!thp_migration_supported ()) + nr_isolated_fast_huge_pages -= nr_exchange_pages; +else + nr_isolated_fast_huge_pages -= nr_exchange_pages *HPAGE_PMD_NR ; + + goto migrate_out; + }else { +migrate_out:/* Migrate pages to slow node */ /* No multi-threaded migration for base pages */ nr_isolated_fast_base_pages -=
@@ -347,6 +519,7 @@ static int do_mm_manage(struct task_struct *p, struct mm_struct *mm, nr_isolated_fast_huge_pages -=migrate_to_node (&fast_huge_page_list, slow_nid, mode, migration_batch_size); + } } if (nr_isolated_fast_base_pages != ULONG_MAX &&
Signature · 1 line
2.7.4
Michal Hocko
maintainerQuoted 11 lines · Zi Yan
From: Zi Yan <[email protected]>
Thanks to Dave Hansen's patches, which make PMEM as part of memory as NUMA nodes. How to use PMEM along with normal DRAM remains an open problem. There are several patchsets posted on the mailing list, proposing to use page migration to move pages between PMEM and DRAM using Linux page replacement policy [1,2,3]. There are some important problems not addressed in these patches:
1. The page migration in Linux does not provide high enough throughput for us to fully exploit PMEM or other use cases.
2. Linux page replacement is running too infrequent to distinguish hot and cold pages.
[...]
Quoted 1 line
33 files changed, 4261 insertions(+), 162 deletions(-)
For a patch _this_ large you should really start with a real world usecasing hitting bottlenecks with the current implementation. Should microbenchmarks can trigger bottlenecks much easier but do real application do the same? Please give us some numbers.
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 96 lines · Zi Yan
From: Zi Yan <[email protected]>
Thanks to Dave Hansen's patches, which make PMEM as part of memory as NUMA nodes. How to use PMEM along with normal DRAM remains an open problem. There are several patchsets posted on the mailing list, proposing to use page migration to move pages between PMEM and DRAM using Linux page replacement policy [1,2,3]. There are some important problems not addressed in these patches:
1. The page migration in Linux does not provide high enough throughput for us to fully exploit PMEM or other use cases.
2. Linux page replacement is running too infrequent to distinguish hot and cold pages.I am trying to attack the problems with this patch series. This is not a final solution, but I would like to gather more feedback and comments from the mailing list.
Page migration throughput problem
====For example, in my recent email [4], I gave the page migration throughput numbers for different page migrations, none of which can achieve > 2.5GB/s throughput (the throughput is measured around kernel functions:
migrate_pages()andmigrate_page_copy()):| migrate_pages() | migrate_page_copy() migrating single 4KB page: | 0.312GB/s | 1.385GB/s migrating 512 4KB pages: | 0.854GB/s | 1.983GB/s migrating single 2MB THP: | 2.387GB/s | 2.481GB/sIn reality, microbenchmarks show that Intel PMEM can provide ~65GB/s read throughput and ~16GB/s write throughput [5], which are much higher than the throughput achieved by Linux page migration.
In addition, it is also desirable to use page migration to move data between high-bandwidth memory and DRAM, like IBM Summit, which exposes high-performance GPU memories as NUMA nodes [6]. This requires even higher page migration throughput.
In this patch series, I propose four different ways of improving page migration throughput (mostly on 2MB THP migration):
1. multi-threaded page migration: Patch 03 to 06.
2. DMA-based (using Intel IOAT DMA) page migration: Patch 07 and 08.
3. concurrent (batched) page migration: Patch 09, 10, and 11.
4. exchange pages: Patch 12 to 17. (This is a repost of part of [7])Here are some throughput numbers showing clear throughput improvements on a two-socket NUMA machine with two Xeon E5-2650 v3 @ 2.30GHz and a 19.2GB/s bandwidth QPI link (the same machine as mentioned in [4]):
| migrate_pages() | migrate_page_copy() => migrating single 2MB THP | 2.387GB/s | 2.481GB/s 2-thread single THP migration | 3.478GB/s | 3.704GB/s 4-thread single THP migration | 5.474GB/s | 6.054GB/s 8-thread single THP migration | 7.846GB/s | 9.029GB/s 16-thread single THP migration | 7.423GB/s | 8.464GB/s 16-ch. DMA single THP migration | 4.322GB/s | 4.536GB/s2-thread 16-THP migration | 3.610GB/s | 3.838GB/s 2-thread 16-THP batched migration | 4.138GB/s | 4.344GB/s 4-thread 16-THP migration | 6.385GB/s | 7.031GB/s 4-thread 16-THP batched migration | 7.382GB/s | 8.072GB/s 8-thread 16-THP migration | 8.039GB/s | 9.029GB/s 8-thread 16-THP batched migration | 9.023GB/s | 10.056GB/s 16-thread 16-THP migration | 8.137GB/s | 9.137GB/s 16-thread 16-THP batched migration | 9.907GB/s | 11.175GB/s1-thread 16-THP exchange | 4.135GB/s | 4.225GB/s 2-thread 16-THP batched exchange | 7.061GB/s | 7.325GB/s 4-thread 16-THP batched exchange | 9.729GB/s | 10.237GB/s 8-thread 16-THP batched exchange | 9.992GB/s | 10.533GB/s16-thread 16-THP batched exchange | 9.520GB/s | 10.056GB/s
=> migrating 512 4KB pages | 0.854GB/s | 1.983GB/s
1-thread 512-4KB batched exchange | 1.271GB/s | 3.433GB/s 2-thread 512-4KB batched exchange | 1.240GB/s | 3.190GB/s 4-thread 512-4KB batched exchange | 1.255GB/s | 3.823GB/s 8-thread 512-4KB batched exchange | 1.336GB/s | 3.921GB/s16-thread 512-4KB batched exchange | 1.334GB/s | 3.897GB/s
Concerns were raised on how to avoid CPU resource competition between page migration and user applications and have power awareness. Daniel Jordan recently posted a multi-threaded ktask patch series could be a solution [8].
Infrequent page list update problem
====Current page lists are updated by calling
shrink_list()when memory pressure comes, which might not be frequent enough to keep track of hot and cold pages. Because all pages are on active lists at the first timeshrink_list()is called and the reference bit on the pages might not reflect the up to date access status of these pages. But we also do not want to periodically shrink the global page lists, which adds unnecessary overheads to the whole system. So I propose to actively shrink page lists on the memcg we are interested in.
Quoted 5 lines
Patch 18 to 25 add a new system call to shrink page lists on given application's memcg and migrate pages between two NUMA nodes. It isolates the impact from the rest of the system. To share DRAM among different applications, Patch 18 and 19 add per-node memcg size limit, so you can limit the memory usage for particular NUMA node(s).
This sounds a little bit confusing to me. Is it totally user's decision about when to call the syscall to shrink page lists? But, how would user know when is a good timing? Could you please elaborate the usecase?
Thanks,
Yang
Quoted 98 lines
Patch structure
====
1. multi-threaded page migration: Patch 01 to 06.
2. DMA-based (using Intel IOAT DMA) page migration: Patch 07 and 08.
3. concurrent (batched) page migration: Patch 09, 10, and 11.
4. exchange pages: Patch 12 to 17. (This is a repost of part of [7])
5. per-node size limit in memcg: Patch 18 and 19.
6. actively shrink page lists and perform page migration in given memcg: Patch 20 to 25.Any comment is welcome.
[1]: https://lore.kernel.org/linux-mm/[email protected]/
[2]: https://lore.kernel.org/linux-mm/[email protected]/
[3]: https://lore.kernel.org/linux-mm/[email protected]/
[4]: https://lore.kernel.org/linux-mm/[email protected]/
[5]: https://www.storagereview.com/supermicro_superserver_with_intel_optane_dc_persistent_memory_first_look_review
[6]: https://www.ibm.com/thought-leadership/summit-supercomputer/
[7]: https://lore.kernel.org/linux-mm/[email protected]/
[8]: https://lore.kernel.org/linux-mm/[email protected]/Zi Yan (25):
mm: migrate: Changemigrate_modeto support combination migration
modes.
mm: migrate: Add mode parameter to support future page copy routines.
mm: migrate: Add a multi-threaded page migration function.
mm: migrate: Addcopy_page_multithreadintomigrate_pages.
mm: migrate: Add vm.accel_page_copy in sysfs to control page copy
acceleration.
mm: migrate: Make the number of copy threads adjustable via sysctl.
mm: migrate: Addcopy_page_dmato use DMA Engine to copy pages.
mm: migrate: Addcopy_page_dmaintomigrate_page_copy.
mm: migrate: Addcopy_page_lists_dma_alwaysto support copy a list of
pages.
mm: migrate:copy_page_lists_mt()to copy a page list using
multi-threads.
mm: migrate: Add concurrent page migration intomove_pagessyscall. exchange pages: new page migration mechanism:exchange_pages()exchange pages: add multi-threaded exchange pages. exchange pages: concurrent exchange pages.
exchange pages: exchange anonymous page and file-backed page. exchange page: Add THP exchange support.
exchange page: Addexchange_page()syscall.
memcg: Add per node memory usage&max stats in memcg.
mempolicy: addMPOL_F_MEMCGflag, enforcing memcg memory limit. memory manage: Add memory manage syscall.
mm: moveupdate_lru_sizes()tomm_inline.hfor broader use. memory manage: active/inactive page list manipulation in memcg. memory manage: page migration based page manipulation between NUMA
nodes.
memory manage: limit migration batch size.
memory manage: use exchange pages to memory manage to improve
throughput.arch/x86/entry/syscalls/syscall_64.tbl | 2 + fs/aio.c | 12 +- fs/f2fs/data.c | 6 +- fs/hugetlbfs/inode.c | 4 +- fs/iomap.c | 4 +- fs/ubifs/file.c | 4 +- include/linux/cgroup-defs.h | 1 + include/linux/exchange.h | 27 + include/linux/highmem.h | 3 + include/linux/ksm.h | 4 + include/linux/memcontrol.h | 67 ++ include/linux/migrate.h | 12 +- include/linux/migrate_mode.h | 8 + include/linux/mm_inline.h | 21 + include/linux/sched/coredump.h | 1 + include/linux/sched/sysctl.h | 3 + include/linux/syscalls.h | 10 + include/uapi/linux/mempolicy.h | 9 +- kernel/sysctl.c | 47 + mm/Makefile | 5 + mm/balloon_compaction.c | 2 +- mm/compaction.c | 22 +- mm/copy_page.c | 708 +++++++++++++++ mm/exchange.c | 1560 ++++++++++++++++++++++++++++++++ mm/exchange_page.c | 228 +++++ mm/internal.h | 113 +++ mm/ksm.c | 35 + mm/memcontrol.c | 80 ++ mm/memory_manage.c | 649 +++++++++++++ mm/mempolicy.c | 38 +- mm/migrate.c | 621 ++++++++++++- mm/vmscan.c | 115 +-- mm/zsmalloc.c | 2 +- 33 files changed, 4261 insertions(+), 162 deletions(-)create mode 100644
include/linux/exchange.h
create mode 100644mm/copy_page.c
create mode 100644mm/exchange.c
create mode 100644mm/exchange_page.c
create mode 100644mm/memory_manage.c--
2.7.4
Zi Yan
reviewerQuoted 12 lines
Quoted 12 lines
Infrequent page list update problem
====Current page lists are updated by calling
shrink_list()when memory pressure comes, which might not be frequent enough to keep track of hot and cold pages. Because all pages are on active lists at the first timeshrink_list()is called and the reference bit on the pages might not reflect the up to date access status of these pages. But we also do not want to periodically shrink the global page lists, which adds unnecessary overheads to the whole system. So I propose to actively shrink page lists on the memcg we are interested in.
Quoted 7 lines
Quoted 5 lines
Patch 18 to 25 add a new system call to shrink page lists on given application's memcg and migrate pages between two NUMA nodes. It isolates the impact from the rest of the system. To share DRAM among different applications, Patch 18 and 19 add per-node memcg size limit, so you can limit the memory usage for particular NUMA node(s).
This sounds a little bit confusing to me. Is it totally user's decision about when to call the syscall to shrink page lists? But, how would user know when is a good timing? Could you please elaborate the usecase?
Sure. We would set up a daemon that monitors user applications and calls the syscall to shuffle the page lists for the user applications, although the daemon’s concrete action plan is still under exploration. It might not be ideal but the page access information could be refreshed periodically and page migration would happen on the background of application execution.
On the other hand, if we wait until DRAM is full and use page migration to make room in DRAM for either page promotion or new page allocation, page migration sits on the critical path of application execution. Considering the bandwidth and access latency gaps between DRAM and PMEM are not as large as the gaps between DRAM and SSD, the cost of page migration (4KB/0.312GB/s = 12us or 2MB/2.387GB/s = 818us)might defeat the benefit of using DRAM over PMEM. I just wonder which would be better: waiting for 12us or 818us then reading 4KB or 2MB data in DRAM or directly accessing the data in PMEM without waiting.
Let me know if this makes sense to you.
Thanks.
--
Best Regards,
Yan Zi