Enable parallel page migration
Subsystems (11)
cover Enable parallel page migration
This patch series is base on the work posted by Zi Yan back in
November 2016 (https://lkml.org/lkml/2016/11/22/457) but includes some amount clean up and re-organization. This series depends on THP migration optimization patch series posted by Naoya Horiguchi on 8th November 2016
(https://lwn.net/Articles/705879/). Though Zi Yan has recently reposted V3 of the THP migration patch series (https://lwn.net/Articles/713667/),
this series is yet to be rebased.
Primary motivation behind this patch series is to achieve higher
bandwidth of memory migration when ever possible using multi threaded instead of a single threaded copy. Did all the experiments using a two socket X86 sytsem (Intel(R) Xeon(R) CPU E5-2650). All the experiments here have same allocation size 4K * 100000 (which did not split evenly for the 2MB huge pages). Here are the results.
Vanilla:
Moved 100000 normal pages in 247.000000 msecs 1.544412 GBs Moved 100000 normal pages in 238.000000 msecs 1.602814 GBs Moved 195 huge pages in 252.000000 msecs 1.513769 GBs Moved 195 huge pages in 257.000000 msecs 1.484318 GBs
THP migration improvements:
Moved 100000 normal pages in 302.000000 msecs 1.263145 GBs Moved 100000 normal pages in 262.000000 msecs 1.455991 GBs Moved 195 huge pages in 120.000000 msecs 3.178914 GBs Moved 195 huge pages in 129.000000 msecs 2.957130 GBs
THP migration improvements + Multi threaded page copy:
Moved 100000 normal pages in 1589.000000 msecs 0.240069 GBs ** Moved 100000 normal pages in 1932.000000 msecs 0.197448 GBs ** Moved 195 huge pages in 54.000000 msecs 7.064254 GBs *** Moved 195 huge pages in 86.000000 msecs 4.435694 GBs ***
** Using multi threaded copy can be detrimental to performance if
used for regular pages which are way too small. But then the framework provides the means to use it if some kernel/driver caller or user application wants to use it.
*** These applications have used the new MPOL_MF_MOVE_MT flag while
calling the system calls like mbind() and move_pages().
On POWER8 the improvements are similar when tested with a draft patch which enables migration at PMD level. Not putting out the results here as the kernel is not stable with the that draft patch and crashes some times. We are working on enabling PMD level migration on POWER8 and will test this series out thoroughly when its ready.
Patch Series Description::
Patch 1: Add new parameter to migrate_page_copy and copy_huge_page so
that it can differentiate between when to use single threaded version (MIGRATE_ST) or multi threaded version (MIGRATE_MT).
Patch 2: Make migrate_mode types non-exclusive.
Patch 3: Add copy_pages_mthread function which does the actual multi
threaded copy. This involves splitting the copy work into chunks, selecting threads and submitting copy jobs in the work queues.
Patch 4: Add new migrate mode MIGRATE_MT to be used by higher level
migration functions.
Patch 5: Add new migration flag MPOL_MF_MOVE_MT for migration system
calls to be used in the user space.
Patch 6: Define global mt_page_copy tunable which turns on the multi
threaded page copy no matter what for all migrations on the system.
Outstanding Issues::
Issue 1: The usefulness of the global multi threaded copy tunable i.e
vm.mt_page_copy. It makes sense and helps in validating the framework. Should this be moved to debugfs instead ?
Issue 2: We choose nr_copythreads = 8 as maximum number of threads on
a node can be 8 on any architecture (Which is on POWER8 if I am not missing any other arch which might have equal or more number of threads per node). It just denotes max number of threads and we will be adjusted based on cpumask_weight value on destination node. Can we do better, suggestions ?
Issue 3: Multi threaded page migration works best with threads allocated
at different physical cores, not all in the same hyper-threaded core. Work queues submitted jobs consume scheduler slots from the given thread to execute the copy. This can interfere with scheduling and affect some already running tasks on the system. Should we be looking into arch topology information, scheduler cpu idle details to decide on which threads to use before going for multi threaded copy ? Abort multi threaded copy and fallback to regular copy at times when the parameters are not good ?
Any comments, suggestions are welcome.
Zi Yan (6):
mm/migrate: Add new mode parameter to migrate_page_copy() function mm/migrate: Make migrate_mode types non-exclussive mm/migrate: Add copy_pages_mthread function
mm/migrate: Add new migrate mode MIGRATE_MT
mm/migrate: Add new migration flag MPOL_MF_MOVE_MT for syscalls
sysctl: Add global tunable mt_page_copy
fs/aio.c | 2 +- fs/f2fs/data.c | 2 +- fs/hugetlbfs/inode.c | 2 +- fs/ubifs/file.c | 2 +- include/linux/highmem.h | 2 + include/linux/migrate.h | 6 ++- include/linux/migrate_mode.h | 8 ++-- include/uapi/linux/mempolicy.h | 4 +- kernel/sysctl.c | 10 +++++ mm/Makefile | 2 + mm/compaction.c | 20 +++++----- mm/copy_pages_mthread.c | 87 ++++++++++++++++++++++++++++++++++++++++++ mm/mempolicy.c | 7 +++- mm/migrate.c | 81 +++++++++++++++++++++++++++------------ 14 files changed, 190 insertions(+), 45 deletions(-) create mode 100644 mm/copy_pages_mthread.c
Signature · 7 lines
2.9.3 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 1/6 mm/migrate: Add new mode parameter to migrate_page_copy() function
From: Zi Yan <[email protected]>
This is a prerequisite change required to make page migration framewok copy in different modes like the default single threaded or the new multi threaded one yet to be introduced in follow up patches. This does not change any existing functionality. Only migrate_page_copy() and copy_huge_page() function's signatures are affected.
--- fs/aio.c | 2 +- fs/f2fs/data.c | 2 +- fs/hugetlbfs/inode.c | 2 +- fs/ubifs/file.c | 2 +- include/linux/migrate.h | 6 ++++-- include/linux/migrate_mode.h | 1 + mm/migrate.c | 14 ++++++++------ 7 files changed, 17 insertions(+), 12 deletions(-)
fs/aio.c +1 -1
@@ -418,7 +418,7 @@ static int aio_migratepage(struct address_space *mapping, struct page *new,* events from being lost. */ spin_lock_irqsave (&ctx->completion_lock, flags); -migrate_page_copy (new, old); +migrate_page_copy (new, old,MIGRATE_ST );BUG_ON (ctx->ring_pages[idx] != old); ctx->ring_pages[idx] = new;spin_unlock_irqrestore (&ctx->completion_lock, flags);
fs/f2fs/data.c +1 -1
@@ -1997,7 +1997,7 @@ int f2fs_migrate_page(struct address_space *mapping,SetPagePrivate (newpage);set_page_private (newpage,page_private (page)); -migrate_page_copy (newpage, page); +migrate_page_copy (newpage, page,MIGRATE_ST );return MIGRATEPAGE_SUCCESS ; }
fs/hugetlbfs/inode.c +1 -1
@@ -850,7 +850,7 @@ static int hugetlbfs_migrate_page(struct address_space *mapping, rc =migrate_huge_page_move_mapping (mapping, newpage, page);if (rc !=MIGRATEPAGE_SUCCESS )return rc; -migrate_page_copy (newpage, page); +migrate_page_copy (newpage, page,MIGRATE_ST );return MIGRATEPAGE_SUCCESS ; }
fs/ubifs/file.c +1 -1
@@ -1482,7 +1482,7 @@ static int ubifs_migrate_page(struct address_space *mapping, SetPagePrivate(newpage); } -migrate_page_copy (newpage, page); +migrate_page_copy (newpage, page,MIGRATE_ST );return MIGRATEPAGE_SUCCESS ; }#endif
include/linux/migrate.h +4 -2
@@ -42,7 +42,8 @@ extern void putback_movable_page(struct page *page);extern int migrate_prep (void );extern int migrate_prep_local (void ); -extern void migrate_page_copy (struct page *newpage,struct page *page); +extern void migrate_page_copy (struct page *newpage,struct page *page, +enum migrate_mode mode);extern int migrate_huge_page_move_mapping (struct address_space *mapping,struct page *newpage,struct page *page); extern int migrate_page_move_mapping(struct address_space *mapping,
@@ -61,7 +62,8 @@ static inline int migrate_prep(void) { return -ENOSYS; }static inline int migrate_prep_local (void ) {return -ENOSYS ; }static inline void migrate_page_copy (struct page *newpage, -struct page *page) {} +struct page *page, +enum migrate_mode mode) {}static inline int migrate_huge_page_move_mapping (struct address_space *mapping,struct page *newpage,struct page *page)
include/linux/migrate_mode.h +1 -0
@@ -11,6 +11,7 @@ enum migrate_mode {MIGRATE_ASYNC ,MIGRATE_SYNC_LIGHT ,MIGRATE_SYNC , +MIGRATE_ST };#endif /* MIGRATE_MODE_H_INCLUDED */
mm/migrate.c +8 -6
@@ -589,7 +589,8 @@ static void __copy_gigantic_page(struct page *dst, struct page *src, } } -static void copy_huge_page (struct page *dst,struct page *src) +static void copy_huge_page (struct page *dst,struct page *src, +enum migrate_mode mode) {int i;int nr_pages;
@@ -618,12 +619,13 @@ static void copy_huge_page(struct page *dst, struct page *src)/* * Copy the page to its new location */ -void migrate_page_copy (struct page *newpage,struct page *page) +void migrate_page_copy (struct page *newpage,struct page *page, +enum migrate_mode mode) {int cpupid;if (PageHuge (page) ||PageTransHuge (page)) -copy_huge_page (newpage, page); +copy_huge_page (newpage, page, mode);else copy_highpage (newpage, page);
@@ -705,7 +707,7 @@ int migrate_page(struct address_space *mapping,if (rc !=MIGRATEPAGE_SUCCESS )return rc; -migrate_page_copy (newpage, page); +migrate_page_copy (newpage, page, mode);return MIGRATEPAGE_SUCCESS ; }EXPORT_SYMBOL (migrate_page);
@@ -755,7 +757,7 @@ int buffer_migrate_page(struct address_space *mapping,SetPagePrivate (newpage); -migrate_page_copy (newpage, page); +migrate_page_copy (newpage, page,MIGRATE_ST ); bh = head; do {
@@ -1968,7 +1970,7 @@ int migrate_misplaced_transhuge_page(struct mm_struct *mm,/* anon mapping, we can simply copy page->mapping to the new page: */ new_page->mapping = page->mapping; new_page->index = page->index; -migrate_page_copy (new_page, page); +migrate_page_copy (new_page, page,MIGRATE_ST );WARN_ON (PageLRU (new_page));/* Recheck the target PMD */
Signature · 7 lines
2.9.3 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 1/6 mm/migrate: Add new mode parameter to migrate_page_copy() function
From: Zi Yan <[email protected]>
This is a prerequisite change required to make page migration framewok copy in different modes like the default single threaded or the new multi threaded one yet to be introduced in follow up patches. This does not change any existing functionality. Only migrate_page_copy() and copy_huge_page() function's signatures are affected.
---
* Updated include/linux/migrate_mode.h comment as per Naoya
fs/aio.c | 2 +- fs/f2fs/data.c | 2 +- fs/hugetlbfs/inode.c | 2 +- fs/ubifs/file.c | 2 +- include/linux/migrate.h | 6 ++++-- include/linux/migrate_mode.h | 2 ++ mm/migrate.c | 14 ++++++++------ 7 files changed, 18 insertions(+), 12 deletions(-)
fs/aio.c +1 -1
@@ -418,7 +418,7 @@ static int aio_migratepage(struct address_space *mapping, struct page *new,* events from being lost. */ spin_lock_irqsave (&ctx->completion_lock, flags); -migrate_page_copy (new, old); +migrate_page_copy (new, old,MIGRATE_ST );BUG_ON (ctx->ring_pages[idx] != old); ctx->ring_pages[idx] = new;spin_unlock_irqrestore (&ctx->completion_lock, flags);
fs/f2fs/data.c +1 -1
@@ -1997,7 +1997,7 @@ int f2fs_migrate_page(struct address_space *mapping,SetPagePrivate (newpage);set_page_private (newpage,page_private (page)); -migrate_page_copy (newpage, page); +migrate_page_copy (newpage, page,MIGRATE_ST );return MIGRATEPAGE_SUCCESS ; }
fs/hugetlbfs/inode.c +1 -1
@@ -850,7 +850,7 @@ static int hugetlbfs_migrate_page(struct address_space *mapping, rc =migrate_huge_page_move_mapping (mapping, newpage, page);if (rc !=MIGRATEPAGE_SUCCESS )return rc; -migrate_page_copy (newpage, page); +migrate_page_copy (newpage, page,MIGRATE_ST );return MIGRATEPAGE_SUCCESS ; }
fs/ubifs/file.c +1 -1
@@ -1482,7 +1482,7 @@ static int ubifs_migrate_page(struct address_space *mapping, SetPagePrivate(newpage); } -migrate_page_copy (newpage, page); +migrate_page_copy (newpage, page,MIGRATE_ST );return MIGRATEPAGE_SUCCESS ; }#endif
include/linux/migrate.h +4 -2
@@ -42,7 +42,8 @@ extern void putback_movable_page(struct page *page);extern int migrate_prep (void );extern int migrate_prep_local (void ); -extern void migrate_page_copy (struct page *newpage,struct page *page); +extern void migrate_page_copy (struct page *newpage,struct page *page, +enum migrate_mode mode);extern int migrate_huge_page_move_mapping (struct address_space *mapping,struct page *newpage,struct page *page); extern int migrate_page_move_mapping(struct address_space *mapping,
@@ -61,7 +62,8 @@ static inline int migrate_prep(void) { return -ENOSYS; }static inline int migrate_prep_local (void ) {return -ENOSYS ; }static inline void migrate_page_copy (struct page *newpage, -struct page *page) {} +struct page *page, +enum migrate_mode mode) {}static inline int migrate_huge_page_move_mapping (struct address_space *mapping,struct page *newpage,struct page *page)
include/linux/migrate_mode.h +2 -0
@@ -6,11 +6,13 @@* on most operations but not ->writepage as the potential stall time * is too significant * MIGRATE_SYNC will block when migrating pages +* MIGRATE_ST will use single thread when migrating pages */ enum migrate_mode {MIGRATE_ASYNC ,MIGRATE_SYNC_LIGHT ,MIGRATE_SYNC , +MIGRATE_ST };#endif /* MIGRATE_MODE_H_INCLUDED */
mm/migrate.c +8 -6
@@ -631,7 +631,8 @@ static void __copy_gigantic_page(struct page *dst, struct page *src, } } -static void copy_huge_page (struct page *dst,struct page *src) +static void copy_huge_page (struct page *dst,struct page *src, +enum migrate_mode mode) {int i;int nr_pages;
@@ -660,12 +661,13 @@ static void copy_huge_page(struct page *dst, struct page *src)/* * Copy the page to its new location */ -void migrate_page_copy (struct page *newpage,struct page *page) +void migrate_page_copy (struct page *newpage,struct page *page, +enum migrate_mode mode) {int cpupid;if (PageHuge (page) ||PageTransHuge (page)) -copy_huge_page (newpage, page); +copy_huge_page (newpage, page, mode);else copy_highpage (newpage, page);
@@ -747,7 +749,7 @@ int migrate_page(struct address_space *mapping,if (rc !=MIGRATEPAGE_SUCCESS )return rc; -migrate_page_copy (newpage, page); +migrate_page_copy (newpage, page, mode);return MIGRATEPAGE_SUCCESS ; }EXPORT_SYMBOL (migrate_page);
@@ -797,7 +799,7 @@ int buffer_migrate_page(struct address_space *mapping,SetPagePrivate (newpage); -migrate_page_copy (newpage, page); +migrate_page_copy (newpage, page,MIGRATE_ST ); bh = head; do {
@@ -2029,7 +2031,7 @@ int migrate_misplaced_transhuge_page(struct mm_struct *mm,/* anon mapping, we can simply copy page->mapping to the new page: */ new_page->mapping = page->mapping; new_page->index = page->index; -migrate_page_copy (new_page, page); +migrate_page_copy (new_page, page,MIGRATE_ST );WARN_ON (PageLRU (new_page));/* Recheck the target PMD */
Signature · 7 lines
2.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 2/6 mm/migrate: Make migrate_mode types non-exclusive
In reply to Anshuman KhandualFrom: Zi Yan <[email protected]>
It basically changes the enum declaration from numbers to bit positions so that they can be used in combination which was not the case earlier. No functionality has been changed.
--- include/linux/migrate_mode.h | 8 ++++---- mm/compaction.c | 20 ++++++++++---------- mm/migrate.c | 14 +++++++------- 3 files changed, 21 insertions(+), 21 deletions(-)
include/linux/migrate_mode.h +4 -4
@@ -8,10 +8,10 @@* MIGRATE_SYNC will block when migrating pages */ enum migrate_mode { -MIGRATE_ASYNC , -MIGRATE_SYNC_LIGHT , -MIGRATE_SYNC , -MIGRATE_ST +MIGRATE_ASYNC =1 <<0 , +MIGRATE_SYNC_LIGHT =1 <<1 , +MIGRATE_SYNC =1 <<2 , +MIGRATE_ST =1 <<3 , };#endif /* MIGRATE_MODE_H_INCLUDED */
mm/compaction.c +10 -10
@@ -296,7 +296,7 @@ static void update_pageblock_skip(struct compact_control *cc,if (migrate_scanner) {if (pfn > zone->compact_cached_migrate_pfn[0 ]) zone->compact_cached_migrate_pfn[0 ] = pfn; -if (cc->mode !=MIGRATE_ASYNC && +if (!(cc->mode &MIGRATE_ASYNC ) && pfn > zone->compact_cached_migrate_pfn[1 ]) zone->compact_cached_migrate_pfn[1 ] = pfn; }else {
@@ -329,7 +329,7 @@ static void update_pageblock_skip(struct compact_control *cc,static bool compact_trylock_irqsave (spinlock_t *lock,unsigned long *flags,struct compact_control *cc) { -if (cc->mode ==MIGRATE_ASYNC ) { +if (cc->mode &MIGRATE_ASYNC ) {if (!spin_trylock_irqsave (lock, *flags)) { cc->contended = true;return false;
@@ -370,7 +370,7 @@ static bool compact_unlock_should_abort(spinlock_t *lock, }if (need_resched ()) { -if (cc->mode ==MIGRATE_ASYNC ) { +if (cc->mode &MIGRATE_ASYNC ) { cc->contended = true;return true; }
@@ -393,7 +393,7 @@ static inline bool compact_should_abort(struct compact_control *cc) {/* async compaction aborts if contended */ if (need_resched ()) { -if (cc->mode ==MIGRATE_ASYNC ) { +if (cc->mode &MIGRATE_ASYNC ) { cc->contended = true;return true; }
@@ -688,7 +688,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,*/ while (unlikely (too_many_isolated (zone))) {/* async migration should just abort */ -if (cc->mode ==MIGRATE_ASYNC ) +if (cc->mode &MIGRATE_ASYNC )return 0 ;congestion_wait (BLK_RW_ASYNC ,HZ /10 );
@@ -700,7 +700,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,if (compact_should_abort (cc))return 0 ; -if (cc->direct_compaction && (cc->mode ==MIGRATE_ASYNC )) { +if (cc->direct_compaction && (cc->mode &MIGRATE_ASYNC )) { skip_on_failure = true; next_skip_pfn =block_end_pfn (low_pfn, cc->order); }
@@ -1195,7 +1195,7 @@ static isolate_migrate_t isolate_migratepages(struct zone *zone,struct page *page;const isolate_mode_t isolate_mode = (sysctl_compact_unevictable_allowed ?ISOLATE_UNEVICTABLE :0 ) | - (cc->mode !=MIGRATE_SYNC ?ISOLATE_ASYNC_MIGRATE :0 ); + (!(cc->mode &MIGRATE_SYNC ) ?ISOLATE_ASYNC_MIGRATE :0 );/* * Start at where we last stopped, or beginning of the zone as
@@ -1241,7 +1241,7 @@ static isolate_migrate_t isolate_migratepages(struct zone *zone,* Async compaction is optimistic to see if the minimum amount * of work satisfies the allocation. */ -if (cc->mode ==MIGRATE_ASYNC && +if ((cc->mode &MIGRATE_ASYNC ) && !migrate_async_suitable (get_pageblock_migratetype (page)))continue ;
@@ -1481,7 +1481,7 @@ static enum compact_result compact_zone(struct zone *zone, struct compact_controunsigned long start_pfn = zone->zone_start_pfn;unsigned long end_pfn =zone_end_pfn (zone);const int migratetype =gfpflags_to_migratetype (cc->gfp_mask); -const bool sync = cc->mode !=MIGRATE_ASYNC ; +const bool sync = !(cc->mode &MIGRATE_ASYNC ); ret =compaction_suitable (zone, cc->order, cc->alloc_flags, cc->classzone_idx);
@@ -1577,7 +1577,7 @@ static enum compact_result compact_zone(struct zone *zone, struct compact_contro* order-aligned block, so skip the rest of it. */ if (cc->direct_compaction && - (cc->mode ==MIGRATE_ASYNC )) { + (cc->mode &MIGRATE_ASYNC )) { cc->migrate_pfn =block_end_pfn ( cc->migrate_pfn -1 , cc->order);/* Draining pcplists is useless in this case */
mm/migrate.c +7 -7
@@ -352,7 +352,7 @@ static bool buffer_migrate_lock_buffers(struct buffer_head *head,struct buffer_head *bh = head;/* Simple case, sync compaction */ -if (mode !=MIGRATE_ASYNC ) { +if (!(mode &MIGRATE_ASYNC )) { do { get_bh(bh); lock_buffer(bh);
@@ -453,7 +453,7 @@ int migrate_page_move_mapping(struct address_space *mapping,* the mapping back due to an elevated page count, we would have to * block waiting on other references to be dropped. */ -if (mode ==MIGRATE_ASYNC && head && +if ((mode &MIGRATE_ASYNC ) && head && !buffer_migrate_lock_buffers (head, mode)) {page_ref_unfreeze (page, expected_count);spin_unlock_irq (&mapping->tree_lock);
@@ -739,7 +739,7 @@ int buffer_migrate_page(struct address_space *mapping,* with an IRQ-safe spinlock held. In the sync case, the buffers * need to be locked now */ -if (mode !=MIGRATE_ASYNC ) +if (!(mode &MIGRATE_ASYNC ))BUG_ON (!buffer_migrate_lock_buffers (head, mode));ClearPagePrivate (page);
@@ -821,7 +821,7 @@ static int fallback_migrate_page(struct address_space *mapping, {if (PageDirty (page)) {/* Only writeback pages in full synchronous migration */ -if (mode !=MIGRATE_SYNC ) +if (!(mode &MIGRATE_SYNC ))return -EBUSY ;return writeout (mapping, page); }
@@ -930,7 +930,7 @@ static int __unmap_and_move(struct page *page, struct page *newpage,bool is_lru = !__PageMovable (page);if (!trylock_page (page)) { -if (!force || mode ==MIGRATE_ASYNC ) +if (!force || (mode &MIGRATE_ASYNC )) goto out;/*
@@ -959,7 +959,7 @@ static int __unmap_and_move(struct page *page, struct page *newpage,* the retry loop is too short and in the sync-light case, * the overhead of stalling is too much */ -if (mode !=MIGRATE_SYNC ) { +if (!(mode &MIGRATE_SYNC )) { rc = -EBUSY ; goto out_unlock; }
@@ -1229,7 +1229,7 @@ static int unmap_and_move_huge_page(new_page_t get_new_page,return -ENOMEM ;if (!trylock_page (hpage)) { -if (!force || mode !=MIGRATE_SYNC ) +if (!force || !(mode &MIGRATE_SYNC )) goto out;lock_page (hpage); }
Signature · 7 lines
2.9.3 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 3/6 mm/migrate: Add copy_pages_mthread function
In reply to Anshuman KhandualFrom: Zi Yan <[email protected]>
This change adds a new function copy_pages_mthread to enable multi threaded page copy which can be utilized during migration. This function splits the page copy request into multiple threads which will handle individual chunk and send them as jobs to system_highpri_wq work queue.
--- include/linux/highmem.h | 2 ++ mm/Makefile | 2 ++ mm/copy_pages_mthread.c | 87 +++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 91 insertions(+) create mode 100644 mm/copy_pages_mthread.c
include/linux/highmem.h +2 -0
@@ -236,6 +236,8 @@ static inline void copy_user_highpage(struct page *to, struct page *from,#endif +int copy_pages_mthread (struct page *to,struct page *from,int nr_pages); +static inline void copy_highpage (struct page *to,struct page *from) {char *vfrom, *vto;
mm/Makefile +2 -0
@@ -41,6 +41,8 @@ obj-y := filemap.o mempool.o oom_kill.o \ obj-y += init-mm.o +obj-y += copy_pages_mthread.o + ifdef CONFIG_NO_BOOTMEM obj-y += nobootmem.o else
mm/copy_pages_mthread.c new file +87 -0
87 lines not on this page · raw message
Signature · 7 lines
2.9.3 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
kbuild test robot
build botRe: [PATCH 3/6] mm/migrate: Add copy_pages_mthread function
url: https://github.com/0day-ci/linux/commits/Anshuman-Khandual/Enable-parallel-page-migration/20170217-200523 config: i386-tinyconfig (attached as .config) compiler: gcc-6 (Debian 6.2.0-3) 6.2.0…
Hi Zi,
[auto build test WARNING on linus/master]
[also build test WARNING on v4.10-rc8 next-20170216]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
url: https://github.com/0day-ci/linux/commits/Anshuman-Khandual/Enable-parallel-page-migration/20170217-200523
config: i386-tinyconfig (attached as .config)
compiler: gcc-6 (Debian 6.2.0-3) 6.2.0 20160901 reproduce:
# save the attached .config to linux build tree
make ARCH=i386 All warnings (new ones prefixed by >>):
mm/copy_pages_mthread.c: In function 'copy_pages_mthread':
>> mm/copy_pages_mthread.c:49:10: warning: assignment discards 'const' qualifier from pointer target type [-Wdiscarded-qualifiers]
cpumask = cpumask_of_node(node);
^vim +/const +49 mm/copy_pages_mthread.c
33 static void copythread(struct work_struct *work)
34 {
35 struct copy_info *info = (struct copy_info *) work;
36
37 copy_pages(info->to, info->from, info->chunk_size);
38 }
39
40 int copy_pages_mthread(struct page *to, struct page *from, int nr_pages)
41 {
42 struct cpumask *cpumask;
43 struct copy_info *work_items;
44 char *vto, *vfrom;
45 unsigned long i, cthreads, cpu, node, chunk_size;
46 int cpu_id_list[32] = {0};
47
48 node = page_to_nid(to);
> 49 cpumask = cpumask_of_node(node);
50 cthreads = nr_copythreads;
51 cthreads = min_t(unsigned int, cthreads, cpumask_weight(cpumask));
52 cthreads = (cthreads / 2) * 2;
53 work_items = kcalloc(cthreads, sizeof(struct copy_info), GFP_KERNEL);
54 if (!work_items)
55 return -ENOMEM;
56
57 i = 0;---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
Re: [PATCH 3/6] mm/migrate: Add copy_pages_mthread function
Quoted 10 lines · kbuild test robot
Hi Zi,
[auto build test WARNING on linus/master]
[also build test WARNING on v4.10-rc8 next-20170216]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]url: https://github.com/0day-ci/linux/commits/Anshuman-Khandual/Enable-parallel-page-migration/20170217-200523
config: i386-tinyconfig (attached as .config)
compiler: gcc-6 (Debian 6.2.0-3) 6.2.0 20160901 reproduce:
Quoted 8 lines
# save the attached .config to linux build tree make ARCH=i386All warnings (new ones prefixed by >>):
mm/copy_pages_mthread.c: In function 'copy_pages_mthread':Quoted 1 line
Quoted 1 line
mm/copy_pages_mthread.c:49:10: warning: assignment discards 'const' qualifier from pointer target type [-Wdiscarded-qualifiers]cpumask = cpumask_of_node(node);
My bad. This fixes the above warning. Will fix it up next time around.
mm/copy_pages_mthread.c +0 -0
@@ -46,7 +46,7 @@ int copy_pages_mthread(struct page *to, struct page
*from, int nr_pages)
int cpu_id_list[32] = {0};
node = page_to_nid(to);
- cpumask = cpumask_of_node(node);
+ cpumask = (struct cpumask *) cpumask_of_node(node);
cthreads = nr_copythreads;
cthreads = min_t(unsigned int, cthreads, cpumask_weight(cpumask));
cthreads = (cthreads / 2) * 2;--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 3/6 mm/migrate: Add copy_pages_mthread function
In reply to Anshuman KhandualFrom: Zi Yan <[email protected]>
This change adds a new function copy_pages_mthread to enable multi threaded page copy which can be utilized during migration. This function splits the page copy request into multiple threads which will handle individual chunk and send them as jobs to system_highpri_wq work queue.
---
* Updated cthread calculations, taking care of divide by zero issues, picking up the right single thread, defining NR_COPYTHREADS, fixing the build problem on i386 etc.
include/linux/highmem.h | 2 ++ mm/Makefile | 2 ++ mm/copy_pages_mthread.c | 95 +++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 99 insertions(+) create mode 100644 mm/copy_pages_mthread.c
include/linux/highmem.h +2 -0
@@ -236,6 +236,8 @@ static inline void copy_user_highpage(struct page *to, struct page *from,#endif +int copy_pages_mthread (struct page *to,struct page *from,int nr_pages); +static inline void copy_highpage (struct page *to,struct page *from) {char *vfrom, *vto;
mm/Makefile +2 -0
@@ -41,6 +41,8 @@ obj-y := filemap.o mempool.o oom_kill.o \ obj-y += init-mm.o +obj-y += copy_pages_mthread.o + ifdef CONFIG_NO_BOOTMEM obj-y += nobootmem.o else
mm/copy_pages_mthread.c new file +95 -0
95 lines not on this page · raw message
Signature · 7 lines
2.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 4/6 mm/migrate: Add new migrate mode MIGRATE_MT
In reply to Anshuman KhandualFrom: Zi Yan <[email protected]>
This change adds a new migration mode called MIGRATE_MT to enable multi threaded page copy implementation inside copy_huge_page() function by selectively calling copy_pages_mthread() when requested. But it still falls back using the regular page copy mechanism instead the previous multi threaded attempt fails. It also attempts multi threaded copy for regular pages.
--- include/linux/migrate_mode.h | 1 + mm/migrate.c | 25 ++++++++++++++++++------- 2 files changed, 19 insertions(+), 7 deletions(-)
include/linux/migrate_mode.h +1 -0
@@ -12,6 +12,7 @@ enum migrate_mode {MIGRATE_SYNC_LIGHT =1 <<1 ,MIGRATE_SYNC =1 <<2 ,MIGRATE_ST =1 <<3 , + MIGRATE_MT = 1<<4, };#endif /* MIGRATE_MODE_H_INCLUDED */
mm/migrate.c +18 -7
@@ -594,6 +594,7 @@ static void copy_huge_page(struct page *dst, struct page *src, {int i;int nr_pages; +int rc = -EFAULT ;if (PageHuge (src)) {/* hugetlbfs page */
@@ -610,10 +611,14 @@ static void copy_huge_page(struct page *dst, struct page *src, nr_pages =hpage_nr_pages (src); } -for (i =0 ; i < nr_pages; i++) { -cond_resched (); -copy_highpage (dst + i, src + i); - } +if (mode &MIGRATE_MT ) + rc =copy_pages_mthread (dst, src, nr_pages); + +if (rc) +for (i =0 ; i < nr_pages; i++) { +cond_resched (); +copy_highpage (dst + i, src + i); + } }/*
@@ -624,10 +629,16 @@ void migrate_page_copy(struct page *newpage, struct page *page, {int cpupid; -if (PageHuge (page) ||PageTransHuge (page)) +if (PageHuge (page) ||PageTransHuge (page)) {copy_huge_page (newpage, page, mode); -else -copy_highpage (newpage, page); + }else { +if (mode &MIGRATE_MT ) { +if (copy_pages_mthread (newpage, page,1 )) +copy_highpage (newpage, page); + }else { +copy_highpage (newpage, page); + } + }if (PageError (page))SetPageError (newpage);
Signature · 7 lines
2.9.3 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 5/6 mm/migrate: Add new migration flag MPOL_MF_MOVE_MT for syscalls
In reply to Anshuman KhandualFrom: Zi Yan <[email protected]>
This change adds a new mode flag MPOL_MF_MOVE_MT for migration system calls like move_pages() and mbind() which indicates request for using the multi threaded copy method.
--- include/uapi/linux/mempolicy.h | 4 +++- mm/mempolicy.c | 7 ++++++- mm/migrate.c | 14 ++++++++++---- 3 files changed, 19 insertions(+), 6 deletions(-)
include/uapi/linux/mempolicy.h +3 -1
@@ -53,10 +53,12 @@ enum mpol_rebind_step {#define MPOL_MF_MOVE_ALL (1<<2)/* Move every page to conform to policy */ #define MPOL_MF_LAZY (1<<3)/* Modifies '_MOVE: lazy migrate on fault */ #define MPOL_MF_INTERNAL (1<<4)/* Internal flags start here */ +#define MPOL_MF_MOVE_MT (1<<6)/* Use multi-threaded page copy routine */ #define MPOL_MF_VALID (MPOL_MF_STRICT | \ MPOL_MF_MOVE | \ - MPOL_MF_MOVE_ALL) + MPOL_MF_MOVE_ALL | \ + MPOL_MF_MOVE_MT)/* * Internal flags that share the struct mempolicy flags word with
mm/mempolicy.c +6 -1
@@ -1244,9 +1244,14 @@ static long do_mbind(unsigned long start, unsigned long len,int nr_failed =0 ;if (!list_empty (&pagelist)) { +enum migrate_mode mode =MIGRATE_SYNC ; + +if (flags &MPOL_MF_MOVE_MT ) + mode |=MIGRATE_MT ; +WARN_ON_ONCE (flags &MPOL_MF_LAZY ); nr_failed =migrate_pages (&pagelist, new_page,NULL , - start,MIGRATE_SYNC ,MR_MEMPOLICY_MBIND ); + start, mode,MR_MEMPOLICY_MBIND );if (nr_failed)putback_movable_pages (&pagelist); }
mm/migrate.c +10 -4
@@ -1429,11 +1429,16 @@ static struct page *new_page_node(struct page *p, unsigned long private,*/ static int do_move_page_to_node_array (struct mm_struct *mm,struct page_to_node *pm, -int migrate_all) +int migrate_all, +int migrate_use_mt) {int err;struct page_to_node *pp;LIST_HEAD (pagelist); +enum migrate_mode mode =MIGRATE_SYNC ; + +if (migrate_use_mt) + mode |=MIGRATE_MT ;down_read (&mm->mmap_sem);
@@ -1501,7 +1506,7 @@ static int do_move_page_to_node_array(struct mm_struct *mm, err =0 ;if (!list_empty (&pagelist)) { err =migrate_pages (&pagelist, new_page_node,NULL , - (unsigned long )pm,MIGRATE_SYNC ,MR_SYSCALL ); + (unsigned long )pm, mode,MR_SYSCALL );if (err)putback_movable_pages (&pagelist); }
@@ -1578,7 +1583,8 @@ static int do_pages_move(struct mm_struct *mm, nodemask_t task_nodes,/* Migrate this chunk */ err =do_move_page_to_node_array (mm, pm, - flags &MPOL_MF_MOVE_ALL ); + flags &MPOL_MF_MOVE_ALL , + flags &MPOL_MF_MOVE_MT );if (err <0 ) goto out_pm;
@@ -1685,7 +1691,7 @@ SYSCALL_DEFINE6(move_pages, pid_t, pid, unsigned long, nr_pages,nodemask_t task_nodes;/* Check flags */ -if (flags & ~(MPOL_MF_MOVE |MPOL_MF_MOVE_ALL )) +if (flags & ~(MPOL_MF_MOVE |MPOL_MF_MOVE_ALL |MPOL_MF_MOVE_MT ))return -EINVAL ;if ((flags &MPOL_MF_MOVE_ALL ) && !capable (CAP_SYS_NICE ))
Signature · 7 lines
2.9.3 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 5/6 mm/migrate: Add new migration flag MPOL_MF_MOVE_MT for syscalls
From: Zi Yan <[email protected]>
This change adds a new mode flag MPOL_MF_MOVE_MT for migration system calls like move_pages() and mbind() which indicates request for using the multi threaded copy method.
---
* Updated include/linux/migrate_mode.h comment as per Naoya
include/uapi/linux/mempolicy.h | 4 +++- mm/mempolicy.c | 7 ++++++- mm/migrate.c | 14 ++++++++++---- 3 files changed, 19 insertions(+), 6 deletions(-)
include/uapi/linux/mempolicy.h +3 -1
@@ -53,10 +53,12 @@ enum mpol_rebind_step {#define MPOL_MF_MOVE_ALL (1<<2)/* Move every page to conform to policy */ #define MPOL_MF_LAZY (1<<3)/* Modifies '_MOVE: lazy migrate on fault */ #define MPOL_MF_INTERNAL (1<<4)/* Internal flags start here */ +#define MPOL_MF_MOVE_MT (1<<6)/* Use multi-threaded page copy routine */ #define MPOL_MF_VALID (MPOL_MF_STRICT | \ MPOL_MF_MOVE | \ - MPOL_MF_MOVE_ALL) + MPOL_MF_MOVE_ALL | \ + MPOL_MF_MOVE_MT)/* * Internal flags that share the struct mempolicy flags word with
mm/mempolicy.c +6 -1
@@ -1300,9 +1300,14 @@ static long do_mbind(unsigned long start, unsigned long len,int nr_failed =0 ;if (!list_empty (&pagelist)) { +enum migrate_mode mode =MIGRATE_SYNC ; + +if (flags &MPOL_MF_MOVE_MT ) + mode |=MIGRATE_MT ; +WARN_ON_ONCE (flags &MPOL_MF_LAZY ); nr_failed =migrate_pages (&pagelist, new_page,NULL , - start,MIGRATE_SYNC ,MR_MEMPOLICY_MBIND ); + start, mode,MR_MEMPOLICY_MBIND );if (nr_failed)putback_movable_pages (&pagelist); }
mm/migrate.c +10 -4
@@ -1481,11 +1481,16 @@ static struct page *new_page_node(struct page *p, unsigned long private,*/ static int do_move_page_to_node_array (struct mm_struct *mm,struct page_to_node *pm, -int migrate_all) +int migrate_all, +int migrate_use_mt) {int err;struct page_to_node *pp;LIST_HEAD (pagelist); +enum migrate_mode mode =MIGRATE_SYNC ; + +if (migrate_use_mt) + mode |=MIGRATE_MT ;down_read (&mm->mmap_sem);
@@ -1562,7 +1567,7 @@ static int do_move_page_to_node_array(struct mm_struct *mm, err =0 ;if (!list_empty (&pagelist)) { err =migrate_pages (&pagelist, new_page_node,NULL , - (unsigned long )pm,MIGRATE_SYNC ,MR_SYSCALL ); + (unsigned long )pm, mode,MR_SYSCALL );if (err)putback_movable_pages (&pagelist); }
@@ -1639,7 +1644,8 @@ static int do_pages_move(struct mm_struct *mm, nodemask_t task_nodes,/* Migrate this chunk */ err =do_move_page_to_node_array (mm, pm, - flags &MPOL_MF_MOVE_ALL ); + flags &MPOL_MF_MOVE_ALL , + flags &MPOL_MF_MOVE_MT );if (err <0 ) goto out_pm;
@@ -1746,7 +1752,7 @@ SYSCALL_DEFINE6(move_pages, pid_t, pid, unsigned long, nr_pages,nodemask_t task_nodes;/* Check flags */ -if (flags & ~(MPOL_MF_MOVE |MPOL_MF_MOVE_ALL )) +if (flags & ~(MPOL_MF_MOVE |MPOL_MF_MOVE_ALL |MPOL_MF_MOVE_MT ))return -EINVAL ;if ((flags &MPOL_MF_MOVE_ALL ) && !capable (CAP_SYS_NICE ))
Signature · 7 lines
2.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 6/6 sysctl: Add global tunable mt_page_copy
In reply to Anshuman KhandualFrom: Zi Yan <[email protected]>
A new global sysctl tunable 'mt_page_copy' is added which will override syscall specific requests and enable multi threaded page copy during all migrations on the system. This tunable is disabled by default.
--- kernel/sysctl.c | 10 ++++++++++ mm/migrate.c | 14 ++++++++++++++ 2 files changed, 24 insertions(+)
kernel/sysctl.c +10 -0
@@ -98,6 +98,7 @@#if defined(CONFIG_SYSCTL )/* External variables not in a header file. */ +extern int mt_page_copy;extern int suid_dumpable;#ifdef CONFIG_COREDUMP extern int core_uses_pid;
@@ -1346,6 +1347,15 @@ static struct ctl_table vm_table[] = { .proc_handler = &hugetlb_mempolicy_sysctl_handler, },#endif + { + .procname = "mt_page_copy", + .data = &mt_page_copy, + .maxlen = sizeof(mt_page_copy), + .mode = 0644, + .proc_handler = proc_dointvec, + .extra1 = &zero, + .extra2 = &one, + }, { .procname = "hugetlb_shm_group", .data = &sysctl_hugetlb_shm_group,
mm/migrate.c +14 -0
@@ -48,6 +48,8 @@#include "internal.h" +int mt_page_copy; +/* * migrate_prep() needs to be called before we start compiling a list of pages * to be migrated using isolate_lru_page(). If scheduling work on other CPUs is
@@ -611,6 +613,9 @@ static void copy_huge_page(struct page *dst, struct page *src, nr_pages =hpage_nr_pages (src); } +if (mt_page_copy) + mode |=MIGRATE_MT ; +if (mode &MIGRATE_MT ) rc =copy_pages_mthread (dst, src, nr_pages);
@@ -629,6 +634,9 @@ void migrate_page_copy(struct page *newpage, struct page *page, {int cpupid; +if (mt_page_copy) + mode |=MIGRATE_MT ; +if (PageHuge (page) ||PageTransHuge (page)) {copy_huge_page (newpage, page, mode); }else {
@@ -695,6 +703,12 @@ void migrate_page_copy(struct page *newpage, struct page *page, }EXPORT_SYMBOL (migrate_page_copy); +static int __init mt_page_copy_init (void ) +{ + mt_page_copy =0 ; +return 0 ; +} +subsys_initcall (mt_page_copy_init);/************************************************************ * Migration functions ***********************************************************/
Signature · 7 lines
2.9.3 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
kbuild test robot
build botRe: [PATCH 6/6] sysctl: Add global tunable mt_page_copy
url: https://github.com/0day-ci/linux/commits/Anshuman-Khandual/Enable-parallel-page-migration/20170217-200523 config: i386-randconfig-a0-02131010 (attached as .config) compiler: gcc-6 (Debian…
Hi Zi,
[auto build test ERROR on linus/master]
[also build test ERROR on v4.10-rc8 next-20170217]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
url: https://github.com/0day-ci/linux/commits/Anshuman-Khandual/Enable-parallel-page-migration/20170217-200523
config: i386-randconfig-a0-02131010 (attached as .config)
compiler: gcc-6 (Debian 6.2.0-3) 6.2.0 20160901 reproduce:
# save the attached .config to linux build tree
make ARCH=i386 All errors (new ones prefixed by >>):
>> kernel/built-in.o:(.data+0x1ec8): undefined reference to `mt_page_copy'
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
Re: [PATCH 6/6] sysctl: Add global tunable mt_page_copy
Quoted 9 lines · kbuild test robot
Hi Zi,
[auto build test ERROR on linus/master]
[also build test ERROR on v4.10-rc8 next-20170217]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]url: https://github.com/0day-ci/linux/commits/Anshuman-Khandual/Enable-parallel-page-migration/20170217-200523
config: i386-randconfig-a0-02131010 (attached as .config)
compiler: gcc-6 (Debian 6.2.0-3) 6.2.0 20160901
Though I dont have the same compiler, I am unable to reproduce this build failure exactly. The build fails but for a different symbol. I have the following gcc version but does it really make a difference with respect to finding the symbol etc ?
gcc (Ubuntu 4.9.2-10ubuntu13) 4.9.2
mm/memory.c: In function ?copy_pmd_range?:mm/memory.c:1002:3: error: implicit declaration of function ?pmd_related? [-Werror=implicit-function-declaration]
if (pmd_related(*src_pmd)) {
^
cc1: some warnings being treated as errors scripts/Makefile.build:294: recipe for target 'mm/memory.o' failed
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [kbuild-all] [PATCH 6/6] sysctl: Add global tunable mt_page_copy
Quoted 7 lines · Anshuman Khandual
Quoted 6 lines · kbuild test robot
Hi Zi,
[auto build test ERROR on linus/master]
[also build test ERROR on v4.10-rc8 next-20170217]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
Quoted 6 lines
Quoted 3 lines
url: https://github.com/0day-ci/linux/commits/Anshuman-Khandual/Enable-parallel-page-migration/20170217-200523
config: i386-randconfig-a0-02131010 (attached as .config)
compiler: gcc-6 (Debian 6.2.0-3) 6.2.0 20160901Though I dont have the same compiler, I am unable to reproduce this build failure exactly. The build fails but for a different symbol.
I think previous "undefined reference to `mt_page_copy'" error is due to kbuild bot didn't set CONFIG_MIGRATION (see attached config in original mail) since it is a randconfig test.
Thanks,
Xiaolong
Quoted 14 lines
I have the following gcc version but does it really make a difference with respect to finding the symbol etc ?
gcc (Ubuntu 4.9.2-10ubuntu13) 4.9.2
mm/memory.c: In function a??copy_pmd_rangea??:mm/memory.c:1002:3: error: implicit declaration of function a??pmd_relateda?? [-Werror=implicit-function-declaration]
if (pmd_related(*src_pmd)) {
^
cc1: some warnings being treated as errorsscripts/Makefile.build:294: recipe for target 'mm/memory.o' failed
Quoted 4 lines
_______________________________________________
kbuild-all mailing list
[email protected]
https://lists.01.org/mailman/listinfo/kbuild-all
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [kbuild-all] [PATCH 6/6] sysctl: Add global tunable mt_page_copy
Quoted 8 lines · Ye Xiaolong
Quoted 7 lines · Anshuman Khandual
Quoted 6 lines · kbuild test robot
Hi Zi,
[auto build test ERROR on linus/master]
[also build test ERROR on v4.10-rc8 next-20170217]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
url: https://github.com/0day-ci/linux/commits/Anshuman-Khandual/Enable-parallel-page-migration/20170217-200523
config: i386-randconfig-a0-02131010 (attached as .config)
compiler: gcc-6 (Debian 6.2.0-3) 6.2.0 20160901Though I dont have the same compiler, I am unable to reproduce this build failure exactly. The build fails but for a different symbol.
I think previous "undefined reference to `mt_page_copy'" error is due to kbuild bot didn't set
CONFIG_MIGRATION(see attached config in original mail) since it is a randconfig test.
If CONFIG_MIGRATION is not set then mm/migrate.c never gets compiled and the symbol 'mt_page_copy' is never exported for kernel/sysctl.c based extern variable to use. Sure, will fix it by keeping all the code in kernel/sysctl.c within CONFIG_MIGRATION config. Thanks for pointing it out.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 17 lines · Anshuman Khandual
This patch series is base on the work posted by Zi Yan back in
November 2016 (https://lkml.org/lkml/2016/11/22/457) but includes some amount clean up and re-organization. This series depends on THP migration optimization patch series posted by Naoya Horiguchi on 8th November 2016
(https://lwn.net/Articles/705879/). Though Zi Yan has recently reposted V3 of the THP migration patch series (https://lwn.net/Articles/713667/),
this series is yet to be rebased.Primary motivation behind this patch series is to achieve higher
bandwidth of memory migration when ever possible using multi threaded instead of a single threaded copy. Did all the experiments using a two socket X86 sytsem (Intel(R) Xeon(R) CPU E5-2650). All the experiments here have same allocation size 4K * 100000 (which did not split evenly for the 2MB huge pages). Here are the results.Vanilla:
Quoted 8 lines
Moved 100000 normal pages in 247.000000 msecs 1.544412 GBs Moved 100000 normal pages in 238.000000 msecs 1.602814 GBs Moved 195 huge pages in 252.000000 msecs 1.513769 GBs Moved 195 huge pages in 257.000000 msecs 1.484318 GBs
THP migration improvements:
Moved 100000 normal pages in 302.000000 msecs 1.263145 GBs
Is there a decrease here for normal pages?
Quoted 7 lines
Moved 100000 normal pages in 262.000000 msecs 1.455991 GBs Moved 195 huge pages in 120.000000 msecs 3.178914 GBs Moved 195 huge pages in 129.000000 msecs 2.957130 GBs
THP migration improvements + Multi threaded page copy:
Moved 100000 normal pages in 1589.000000 msecs 0.240069 GBs **
Ditto?
Quoted 4 lines
Moved 100000 normal pages in 1932.000000 msecs 0.197448 GBs ** Moved 195 huge pages in 54.000000 msecs 7.064254 GBs *** Moved 195 huge pages in 86.000000 msecs 4.435694 GBs ***
Could you also comment on the CPU utilization impact of these patches.
Balbir Singh.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 23 lines · Balbir Singh
Quoted 22 lines · Anshuman Khandual
This patch series is base on the work posted by Zi Yan back in
November 2016 (https://lkml.org/lkml/2016/11/22/457) but includes some amount clean up and re-organization. This series depends on THP migration optimization patch series posted by Naoya Horiguchi on 8th November 2016
(https://lwn.net/Articles/705879/). Though Zi Yan has recently reposted V3 of the THP migration patch series (https://lwn.net/Articles/713667/),
this series is yet to be rebased.Primary motivation behind this patch series is to achieve higher
bandwidth of memory migration when ever possible using multi threaded instead of a single threaded copy. Did all the experiments using a two socket X86 sytsem (Intel(R) Xeon(R) CPU E5-2650). All the experiments here have same allocation size 4K * 100000 (which did not split evenly for the 2MB huge pages). Here are the results.Vanilla:
Moved 100000 normal pages in 247.000000 msecs 1.544412 GBs Moved 100000 normal pages in 238.000000 msecs 1.602814 GBs Moved 195 huge pages in 252.000000 msecs 1.513769 GBs Moved 195 huge pages in 257.000000 msecs 1.484318 GBs
Quoted 5 lines
Quoted 3 lines
THP migration improvements:
Moved 100000 normal pages in 302.000000 msecs 1.263145 GBs
Is there a decrease here for normal pages?
Yeah.
Quoted 10 lines
Quoted 8 lines
Moved 100000 normal pages in 262.000000 msecs 1.455991 GBs Moved 195 huge pages in 120.000000 msecs 3.178914 GBs Moved 195 huge pages in 129.000000 msecs 2.957130 GBs
THP migration improvements + Multi threaded page copy:
Moved 100000 normal pages in 1589.000000 msecs 0.240069 GBs **
Ditto?
Yeah, I have already mentioned about this after these data in the cover letter. This new flag is controlled from user space while invoking the system calls. Users should be careful in using it for scenarios where its useful and avoid it for cases where it hurts.
Quoted 8 lines
Quoted 5 lines
Moved 100000 normal pages in 1932.000000 msecs 0.197448 GBs ** Moved 195 huge pages in 54.000000 msecs 7.064254 GBs *** Moved 195 huge pages in 86.000000 msecs 4.435694 GBs ***
Could you also comment on the CPU utilization impact of these patches.
Yeah, it really makes sense to analyze this impact. I have mentioned about this in the outstanding issues section of the series. But what exactly we need to analyze from CPU utilization impact point of view ? Like whats the probability that the work queue requested jobs will throw some tasks from the run queue and make them starve for some more time ? Could you please give some details on this ?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 41 lines · Anshuman Khandual
Quoted 23 lines · Balbir Singh
Quoted 22 lines · Anshuman Khandual
This patch series is base on the work posted by Zi Yan back in
November 2016 (https://lkml.org/lkml/2016/11/22/457) but includes some amount clean up and re-organization. This series depends on THP migration optimization patch series posted by Naoya Horiguchi on 8th November 2016
(https://lwn.net/Articles/705879/). Though Zi Yan has recently reposted V3 of the THP migration patch series (https://lwn.net/Articles/713667/),
this series is yet to be rebased.Primary motivation behind this patch series is to achieve higher
bandwidth of memory migration when ever possible using multi threaded instead of a single threaded copy. Did all the experiments using a two socket X86 sytsem (Intel(R) Xeon(R) CPU E5-2650). All the experiments here have same allocation size 4K * 100000 (which did not split evenly for the 2MB huge pages). Here are the results.Vanilla:
Moved 100000 normal pages in 247.000000 msecs 1.544412 GBs Moved 100000 normal pages in 238.000000 msecs 1.602814 GBs Moved 195 huge pages in 252.000000 msecs 1.513769 GBs Moved 195 huge pages in 257.000000 msecs 1.484318 GBs
Quoted 5 lines
Quoted 3 lines
THP migration improvements:
Moved 100000 normal pages in 302.000000 msecs 1.263145 GBs
Is there a decrease here for normal pages?
Yeah.
Quoted 9 lines
Quoted 8 lines
Moved 100000 normal pages in 262.000000 msecs 1.455991 GBs Moved 195 huge pages in 120.000000 msecs 3.178914 GBs Moved 195 huge pages in 129.000000 msecs 2.957130 GBs
THP migration improvements + Multi threaded page copy:
Moved 100000 normal pages in 1589.000000 msecs 0.240069 GBs **
Quoted 7 lines
Quoted 1 line
Ditto?
Yeah, I have already mentioned about this after these data in the cover letter. This new flag is controlled from user space while invoking the system calls. Users should be careful in using it for scenarios where its useful and avoid it for cases where it hurts.
Fair enough, I wonder if _MT should be disabled for normal pages and allow only THP migration. I think it might be worth evaluating the overheads
Quoted 10 lines
Quoted 9 lines
Quoted 6 lines
Moved 100000 normal pages in 1932.000000 msecs 0.197448 GBs ** Moved 195 huge pages in 54.000000 msecs 7.064254 GBs *** Moved 195 huge pages in 86.000000 msecs 4.435694 GBs ***
Could you also comment on the CPU utilization impact of these patches.
Quoted 7 lines
Yeah, it really makes sense to analyze this impact. I have mentioned about this in the outstanding issues section of the series. But what exactly we need to analyze from CPU utilization impact point of view ? Like whats the probability that the work queue requested jobs will throw some tasks from the run queue and make them starve for some more time ? Could you please give some details on this ?
I wonder if the CPU utilization is so high that its hurting the CPU (system time) at the cost of increased migration speeds. We may need a trade-off (see my comment above)
Balbir Singh.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 97 lines · Anshuman Khandual
This patch series is base on the work posted by Zi Yan back in
November 2016 (https://lkml.org/lkml/2016/11/22/457) but includes some amount clean up and re-organization. This series depends on THP migration optimization patch series posted by Naoya Horiguchi on 8th November 2016
(https://lwn.net/Articles/705879/). Though Zi Yan has recently reposted V3 of the THP migration patch series (https://lwn.net/Articles/713667/),
this series is yet to be rebased.Primary motivation behind this patch series is to achieve higher
bandwidth of memory migration when ever possible using multi threaded instead of a single threaded copy. Did all the experiments using a two socket X86 sytsem (Intel(R) Xeon(R) CPU E5-2650). All the experiments here have same allocation size 4K * 100000 (which did not split evenly for the 2MB huge pages). Here are the results.Vanilla:
Moved 100000 normal pages in 247.000000 msecs 1.544412 GBs Moved 100000 normal pages in 238.000000 msecs 1.602814 GBs Moved 195 huge pages in 252.000000 msecs 1.513769 GBs Moved 195 huge pages in 257.000000 msecs 1.484318 GBs
THP migration improvements:
Moved 100000 normal pages in 302.000000 msecs 1.263145 GBs Moved 100000 normal pages in 262.000000 msecs 1.455991 GBs Moved 195 huge pages in 120.000000 msecs 3.178914 GBs Moved 195 huge pages in 129.000000 msecs 2.957130 GBs
THP migration improvements + Multi threaded page copy:
Moved 100000 normal pages in 1589.000000 msecs 0.240069 GBs ** Moved 100000 normal pages in 1932.000000 msecs 0.197448 GBs ** Moved 195 huge pages in 54.000000 msecs 7.064254 GBs *** Moved 195 huge pages in 86.000000 msecs 4.435694 GBs ***
** Using multi threaded copy can be detrimental to performance if
used for regular pages which are way too small. But then the framework provides the means to use it if some kernel/driver caller or user application wants to use it.*** These applications have used the new
MPOL_MF_MOVE_MTflag while
calling the system calls likembind()andmove_pages().On POWER8 the improvements are similar when tested with a draft patch which enables migration at PMD level. Not putting out the results here as the kernel is not stable with the that draft patch and crashes some times. We are working on enabling PMD level migration on POWER8 and will test this series out thoroughly when its ready.
Patch Series Description::
Patch 1: Add new parameter to
migrate_page_copyandcopy_huge_pageso
that it can differentiate between when to use single threaded version (MIGRATE_ST) or multi threaded version (MIGRATE_MT).Patch 2: Make
migrate_modetypes non-exclusive.Patch 3: Add
copy_pages_mthreadfunction which does the actual multi
threaded copy. This involves splitting the copy work into chunks, selecting threads and submitting copy jobs in the work queues.Patch 4: Add new migrate mode
MIGRATE_MTto be used by higher level
migration functions.Patch 5: Add new migration flag
MPOL_MF_MOVE_MTfor migration system
calls to be used in the user space.Patch 6: Define global
mt_page_copytunable which turns on the multi
threaded page copy no matter what for all migrations on the system.Outstanding Issues::
Issue 1: The usefulness of the global multi threaded copy tunable i.e
vm.mt_page_copy. It makes sense and helps in validating the framework. Should this be moved to debugfs instead ?Issue 2: We choose
nr_copythreads= 8 as maximum number of threads on
a node can be 8 on any architecture (Which is on POWER8 if I am not missing any other arch which might have equal or more number of threads per node). It just denotes max number of threads and we will be adjusted based oncpumask_weightvalue on destination node. Can we do better, suggestions ?Issue 3: Multi threaded page migration works best with threads allocated
at different physical cores, not all in the same hyper-threaded core. Work queues submitted jobs consume scheduler slots from the given thread to execute the copy. This can interfere with scheduling and affect some already running tasks on the system. Should we be looking into arch topology information, scheduler cpu idle details to decide on which threads to use before going for multi threaded copy ? Abort multi threaded copy and fallback to regular copy at times when the parameters are not good ?
Quoted 1 line
Any comments, suggestions are welcome.
Hello Vlastimil/Michal/Minchan/Mel/Dave,
Apart from the comments from Naoya on a different thread posted by Zi Yan, I did not get any more review comments on this series. Could you please kindly have a look on the over all design and its benefits from page migration performance point of view and let me know your views. Thank you.
+ Naoya Horiguchi <[email protected]>
Regards
Anshuman
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 10 lines · Anshuman Khandual
Quoted 1 line
Any comments, suggestions are welcome.
Hello Vlastimil/Michal/Minchan/Mel/Dave,
Apart from the comments from Naoya on a different thread posted by Zi Yan, I did not get any more review comments on this series. Could you please kindly have a look on the over all design and its benefits from page migration performance point of view and let me know your views. Thank you.
I didn't look into the patches in detail except to get a general feel for how it works and I'm not convinced that it's a good idea at all.
I accept that memory bandwidth utilisation may be higher as a result but consider the impact. THP migrations are relatively rare and when they occur, it's in the context of a single thread. To parallelise the copy, an allocation, kmap and workqueue invocation are required. There may be a long delay before the workqueue item can start which may exceed the time to do a single copy if the CPUs on a node are saturated. Furthermore, a single thread can preempt operations of other unrelated threads and incur CPU cache pollution and future misses on unrelated CPUs. It's compounded by the fact that a high priority system workqueue is used to do the operation, one that is used for CPU hotplug operations and rolling back when a netdevice fails to be registered. It treats a hugepage copy as an essential operation that can preempt all other work which is very questionable.
The series leader has no details on a workload that is bottlenecked by THP migrations and even if it is, the primary question should be *why* THP migrations are so frequent and alleviating that instead of preempting multiple CPUs to do the work.
Signature · 8 lines
Mel Gorman SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 16 lines · Mel Gorman
I didn't look into the patches in detail except to get a general feel for how it works and I'm not convinced that it's a good idea at all.
I accept that memory bandwidth utilisation may be higher as a result but consider the impact. THP migrations are relatively rare and when they occur, it's in the context of a single thread. To parallelise the copy, an allocation, kmap and workqueue invocation are required. There may be a long delay before the workqueue item can start which may exceed the time to do a single copy if the CPUs on a node are saturated. Furthermore, a single thread can preempt operations of other unrelated threads and incur CPU cache pollution and future misses on unrelated CPUs. It's compounded by the fact that a high priority system workqueue is used to do the operation, one that is used for CPU hotplug operations and rolling back when a netdevice fails to be registered. It treats a hugepage copy as an essential operation that can preempt all other work which is very questionable.
Quoted 6 lines
The series leader has no details on a workload that is bottlenecked by THP migrations and even if it is, the primary question should be *why* THP migrations are so frequent and alleviating that instead of preempting multiple CPUs to do the work.
Mel - I sense on going frustration around some of the THP migration, migration acceleration, CDM, and other patches. Here is a 10k foot description that I hope adds to what John & Anshuman have said in other threads.
Vendors are currently providing systems that have both traditional DDR3/4 memory (lets call it 100GB/s) and high bandwidth memory (HBM) (lets call it 1TB/s) within a single system. GPUs have been doing this with HBM on the GPU and DDR on the CPU complex, but they've been attached via PCIe and thus HBM has been GPU private memory. The GPU has managed this memory by effectively mlocking pages on the CPU and copying the data into the GPU while its being computed on and then copying it back to the CPU when CPU faults on trying to touch it or the GPU is done. Because HBM is limited in capacity (10's of GB max) versus DDR3 (100's+ GB), runtimes like Nvidia's unified memory dynamically page memory in and out of the GPU to get the benefits of high bandwidth, while still allowing access a total footprint of system memory. Its effectively page protection based CPU/GPU memory coherence.
PCIe attached GPUs+HBM are the bulk or whats out there today and will continue to be, so there are efforts to try and improve how GPUs (and other devices in the same PCIe boat) interact with -mm given the limitations of PCIe (see HMM).
Jumping to what is essentially a different platform - there will be systems where that same GPU HBM memory is now part of the OS controlled memory (aka NUMA node) because these systems have a cache coherent link attaching them (could be NVLINK, QPI, CAPI, HT, or something else) This HBM zone might have CPU cores in it, it might have GPU cores in it, or an FPGA, its not necessarily GPU specific. NVIDIA has talked about systems that look like this, as has Intel (KNL with flat memory), and there are likely others. Systems like this can be thought of (just for exampke) as 2 NUMA node box where you've got 100GB/s of bandwidth on one node, 1TB/s on the other, connected via some cache coherent link. That link is probably order 100GB/s max too (maybe lower, but certainly not 1TB/s yet).
Cores (CPU/GPU/FPGA) can access either NUMA node via the coherent link (just like a multi-socket CPU box) but you also want to be able to optimize page placement so that hot pages physically get migrated into the HBM node from the DDR node. The expectation is that on such systems either the user, a daemon, or kernel/autonuma is going to be migrating (TH)pages between the NUMA zones to optimize overall system bandwidth/throughput. Because of the 10x discrepancy in memory bandwidth, despite the best paging policies to optimize for page locality in the HBM nodes, pages will often still be moving at a high rate between zones. This differs from a traditional NUMA system where moving a page from one 100GB/s node to the other 100GB/s node has dubious value, like you say.
To your specific question - what workloads benefit from this improved migration throughput and why THPs? We have seen that there can be a 1.7x improvement in GPU perf by improving NVLink page migration bandwidth from 6GB/s->32.5GB/s. In comparison, 4KB page migration on x86 over QPI (today) gets < 100MB/s of throughput even though QPI and NVLink can provide 32GB/s+. We couldn't cripple the link enough to get down to ~100MB/s, but obviously using small base page sizes at 100MB/s of migration throughput would kill performance. So good THP functionality + good migration throughput appear critical to us (and maybe KNL too?).
https://devblogs.nvidia.com/parallelforall/beyond-gpu-memory-limits-unified-memory-pascal/
There's a laundry list of things to make it work well related to THPs, migration policy and eviction, numa zone isolation, etc. We hope building functionality into mm + autonuma is useful for everyone so that less of it needs to live in proprietary runtimes.
Hope that helps with 10k foot view and a specific use case on why at least NVIDIA is very interested in optimizing kernel NUMA+THP functionality.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 24 lines · David Nellans
Quoted 16 lines · Mel Gorman
I didn't look into the patches in detail except to get a general feel for how it works and I'm not convinced that it's a good idea at all.
I accept that memory bandwidth utilisation may be higher as a result but consider the impact. THP migrations are relatively rare and when they occur, it's in the context of a single thread. To parallelise the copy, an allocation, kmap and workqueue invocation are required. There may be a long delay before the workqueue item can start which may exceed the time to do a single copy if the CPUs on a node are saturated. Furthermore, a single thread can preempt operations of other unrelated threads and incur CPU cache pollution and future misses on unrelated CPUs. It's compounded by the fact that a high priority system workqueue is used to do the operation, one that is used for CPU hotplug operations and rolling back when a netdevice fails to be registered. It treats a hugepage copy as an essential operation that can preempt all other work which is very questionable.
Quoted 6 lines
The series leader has no details on a workload that is bottlenecked by THP migrations and even if it is, the primary question should be *why* THP migrations are so frequent and alleviating that instead of preempting multiple CPUs to do the work.
Quoted 5 lines
Mel - I sense on going frustration around some of the THP migration, migration acceleration, CDM, and other patches. Here is a 10k foot description that I hope adds to what John & Anshuman have said in other threads.
Hi David,
I recognise the motivation for some of these patches but disagree on the mechanisms used, more on this later.
Quoted 5 lines
Vendors are currently providing systems that have both traditional DDR3/4 memory (lets call it 100GB/s) and high bandwidth memory (HBM) (lets call it 1TB/s) within a single system. GPUs have been doing this with HBM on the GPU and DDR on the CPU complex, but they've been attached via PCIe and thus HBM has been GPU private memory.
I completely understand although I'd point out that HBM is slightly different in that it could be expressed in terms of a hierarchical node system whereby some nodes migrate to each other -- from faster to slower by a "migrate on LRU reclaim" and from slower to faster with automatic NUMA balancing using sampling. However, HBM is extremely specific and dealing with that is not necessarily compatible with devices that are not coherent.
Quoted 1 line
<SNIP>
Again, I understand the motivation and have no further comment to make. In the interest of trying to be helpful, I'll propose an alternative to this series and expand upon why I think it's problematic.
Quoted 10 lines
the HBM node from the DDR node. The expectation is that on such systems either the user, a daemon, or kernel/autonuma is going to be migrating (TH)pages between the NUMA zones to optimize overall system bandwidth/throughput. Because of the 10x discrepancy in memory bandwidth, despite the best paging policies to optimize for page locality in the HBM nodes, pages will often still be moving at a high rate between zones. This differs from a traditional NUMA system where moving a page from one 100GB/s node to the other 100GB/s node has dubious value, like you say.
Quoted 2 lines
To your specific question - what workloads benefit from this improved migration throughput and why THPs?
So the key potential issue here in my mind is that THP migration is too slow in some cases. What I object to is improving that using a high priority workqueue that potentially starves other CPUs and pollutes their cache which is generally very expensive.
Lets look at the core of what copy_huge_page does in mm/migrate.c which is the function that gets parallelised by the series in question. For a !HIGHMEM system, it's woefully inefficient. Historically, it was an implementation that would work generically which was fine but maybe not for future systems. It was also fine back when hugetlbfs was the only huge page implementation and COW operations were incredibly rare on the grounds due to the risk that they could terminate the process with prejudice.
The function takes a huge page, splits it into PAGE_SIZE chunks, kmap_atomics the source and destination for each PAGE_SIZE chunk and copies it. The parallelised version does one kmap and copies it in chunks assuming the THP is fully mapped and accessible. Fundamentally, this is broken in the generic sense as the kmap is not guaranteed to make the whole page necessary but it happens to work on !highmem systems. What is more important to note is that it's multiple preempt and pagefault enables and disables on a per-page basis that happens 512 times (for THP on x86-64 at least), all of which are expensive operations depending on the kernel config and I suspect that the parallisation is actually masking that stupid overhead.
At the very least, I would have expected an initial attempt of one patch that optimised for !highmem systems to ignore kmap, simply disable preempt (if that is even necessary, I didn't check) and copy a pinned physical->physical page as a single copy without looping on a PAGE_SIZE basis and see how much that gained. Do it initially for THP only and worry about gigantic pages when or if that is a problem.
That would be patch 1 of a series. Maybe that'll be enough, maybe not but I feel it's important to optimise the serialised case as much as possible before considering parallelisation to highlight and justify why it's necessary[1]. If nothing else, what if two CPUs both parallelise a migration at the same time and end up preempting each other? Between that and the workqueue setup, it's potentially much slower than an optimised serial copy.
It would be tempting to experiment but the test case was not even included with the series (maybe it's somewhere else)[2]. While it's obvious how such a test case could be constructed, it feels unnecessary to construct it when it should be in the changelog.
To some extent, CDM suffered from the same pushback. Like this series, it introduced something new, complex and with high maintenance overhead without considering whether the existing mechanisms (cpuset, mempolicies or some combination of both) or out-of tree proposals such as HMM that could be added to and finalised[3].
[1] If nothing else, it would make it clear to the reviewer that additional complexity is 100% justified which is woefully missing in this series.
[2] For complex series that have an alleged performance improvement, it should always be possible to supply a test case that can demonstrate that. Granted, this does not always happen but at least for my own series I have the test case in question automated and can point people to the repository that stores the test case and if necessary, supply instructions on how to reproduce the results.
[3] Preferably with an in-tree user because the lack of such a user was one of the major factors that gave HMM a kicking.
Signature · 8 lines
Mel Gorman SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Hi Mel,
Thanks for pointing out the problems in this patchset.
It was my intern project done in NVIDIA last summer. I only used micro-benchmarks to demonstrate the big memory bandwidth utilization gap between base page migration and THP migration along with serialized page migration vs parallel page migration.
Here are cross-socket serialized page migration results from calling move_pages() syscall:
In x86_64, a Intel two-socket E5-2640v3 box,
single 4KB base page migration takes 62.47 us, using 0.06 GB/s BW, single 2MB THP migration takes 658.54 us, using 2.97 GB/s BW, 512 4KB base page migration takes 1987.38 us, using 0.98 GB/s BW.
In ppc64, a two-socket Power8 box,
single 64KB base page migration takes 49.3 us, using 1.24 GB/s BW, single 16MB THP migration takes 2202.17 us, using 7.10 GB/s BW, 256 64KB base page migration takes 2543.65 us, using 6.14 GB/s BW.
THP migration is not slow at all when compared to a group of equivalent base page migrations.
For 1-thread vs 8-thread THP migration:
In x86_64,
1-thread 2MB THP migration takes 658.54 us, using 2.97 GB/s BW, 8-thread 2MB THP migration takes 227.76 us, using 8.58 GB/s BW.
In ppc64,
1-thread 16MB THP migration takes 2202.17 us, using 7.10 GB/s BW, 8-thread 16MB THP migration takes 1223.87 us, using 12.77 GB/s BW.
This big increase on BW utilization is the motivation of pushing this patchset.
Quoted 5 lines
So the key potential issue here in my mind is that THP migration is too slow in some cases. What I object to is improving that using a high priority workqueue that potentially starves other CPUs and pollutes their cache which is generally very expensive.
I might not completely agree with this. Using a high priority workqueue can guarantee page migration work is done ASAP. Otherwise, we completely lose the speedup brought by parallel page migration, if data copy threads have to wait.
I understand your concern on CPU utilization impact. I think checking CPU utilization and only using idle CPUs could potentially avoid this problem.
Quoted 9 lines
Lets look at the core of what
copy_huge_pagedoes inmm/migrate.cwhich is the function that gets parallelised by the series in question. For a !HIGHMEM system, it's woefully inefficient. Historically, it was an implementation that would work generically which was fine but maybe not for future systems. It was also fine back when hugetlbfs was the only huge page implementation and COW operations were incredibly rare on the grounds due to the risk that they could terminate the process with prejudice.
Quoted 10 lines
The function takes a huge page, splits it into
PAGE_SIZEchunks,kmap_atomicsthe source and destination for eachPAGE_SIZEchunk and copies it. The parallelised version does one kmap and copies it in chunks assuming the THP is fully mapped and accessible. Fundamentally, this is broken in the generic sense as the kmap is not guaranteed to make the whole page necessary but it happens to work on !highmem systems. What is more important to note is that it's multiple preempt and pagefault enables and disables on a per-page basis that happens 512 times (for THP on x86-64 at least), all of which are expensive operations depending on the kernel config and I suspect that the parallisation is actually masking that stupid overhead.
You are right on kmap, I think making this patchset depend on !HIGHMEM can avoid the problem. It might not make sense to kmap potentially 512 base pages to migrate a THP in a system with highmem.
Quoted 7 lines
At the very least, I would have expected an initial attempt of one patch that optimised for !highmem systems to ignore kmap, simply disable preempt (if that is even necessary, I didn't check) and copy a pinned
physical->physicalpage as a single copy without looping on aPAGE_SIZEbasis and see how much that gained. Do it initially for THP only and worry about gigantic pages when or if that is a problem.
I can try this out to show how much improvement we can obtain from existing THP migration, which is shown in the data above.
Quoted 8 lines
That would be patch 1 of a series. Maybe that'll be enough, maybe not but I feel it's important to optimise the serialised case as much as possible before considering parallelisation to highlight and justify why it's necessary[1]. If nothing else, what if two CPUs both parallelise a migration at the same time and end up preempting each other? Between that and the workqueue setup, it's potentially much slower than an optimised serial copy.
Quoted 4 lines
It would be tempting to experiment but the test case was not even included with the series (maybe it's somewhere else)[2]. While it's obvious how such a test case could be constructed, it feels unnecessary to construct it when it should be in the changelog.
Do you mean performing multiple parallel page migrations at the same time and show all the page migration time?
Signature · 2 lines
Best Regards, Yan Zi
Quoted 9 lines · Zi Yan
Hi Mel,
Thanks for pointing out the problems in this patchset.
It was my intern project done in NVIDIA last summer. I only used micro-benchmarks to demonstrate the big memory bandwidth utilization gap between base page migration and THP migration along with serialized page migration vs parallel page migration.
The measurement itself is not a problem. It clearly shows why you were doing it and indicates that it's possible.
Quoted 4 lines
<SNIP>
This big increase on BW utilization is the motivation of pushing this patchset.
As before, I have no problem with the motivation, my problem is with the approach and in particular that the serialised case was not optimised first.
Quoted 8 lines
Quoted 5 lines
So the key potential issue here in my mind is that THP migration is too slow in some cases. What I object to is improving that using a high priority workqueue that potentially starves other CPUs and pollutes their cache which is generally very expensive.
I might not completely agree with this. Using a high priority workqueue can guarantee page migration work is done ASAP.
Yes, but at the cost of stalling other operations that are happening at the same tiime. The series assumes that the migration is definitely the most important operation going on at the moment.
Quoted 4 lines
Otherwise, we completely
lose the speedup brought by parallel page migration, if data copy threads have to wait.
And conversely, if important threads were running on the other CPUs at the time the migration started then they might be equally unhappy.
Quoted 4 lines
I understand your concern on CPU utilization impact. I think checking CPU utilization and only using idle CPUs could potentially avoid this problem.
That will be costly to detect actually. It would require poking into the scheduler core and incurring a number of cache misses for a race-prone operation that may not succeed. Even if you do it, it'll still be brought up that the serialised case should be optimised first.
Quoted 11 lines
Quoted 10 lines
The function takes a huge page, splits it into
PAGE_SIZEchunks,kmap_atomicsthe source and destination for eachPAGE_SIZEchunk and copies it. The parallelised version does one kmap and copies it in chunks assuming the THP is fully mapped and accessible. Fundamentally, this is broken in the generic sense as the kmap is not guaranteed to make the whole page necessary but it happens to work on !highmem systems. What is more important to note is that it's multiple preempt and pagefault enables and disables on a per-page basis that happens 512 times (for THP on x86-64 at least), all of which are expensive operations depending on the kernel config and I suspect that the parallisation is actually masking that stupid overhead.
Quoted 4 lines
You are right on kmap, I think making this patchset depend on !HIGHMEM can avoid the problem. It might not make sense to kmap potentially 512 base pages to migrate a THP in a system with highmem.
One concern I have is that the series benefitted the most by simply batching all those operations even if it was not intended.
Quoted 10 lines
Quoted 6 lines
At the very least, I would have expected an initial attempt of one patch that optimised for !highmem systems to ignore kmap, simply disable preempt (if that is even necessary, I didn't check) and copy a pinned
physical->physicalpage as a single copy without looping on aPAGE_SIZEbasis and see how much that gained. Do it initially for THP only and worry about gigantic pages when or if that is a problem.I can try this out to show how much improvement we can obtain from existing THP migration, which is shown in the data above.
It would be important to do so. There would need to be absolute proof that parallelisation is required and even then the concerns about interfering with workloads on other CPUs is not going to be easy to handle.
Quoted 7 lines
Quoted 7 lines
That would be patch 1 of a series. Maybe that'll be enough, maybe not but I feel it's important to optimise the serialised case as much as possible before considering parallelisation to highlight and justify why it's necessary[1]. If nothing else, what if two CPUs both parallelise a migration at the same time and end up preempting each other? Between that and the workqueue setup, it's potentially much slower than an optimised serial copy.
Quoted 7 lines
Quoted 4 lines
It would be tempting to experiment but the test case was not even included with the series (maybe it's somewhere else)[2]. While it's obvious how such a test case could be constructed, it feels unnecessary to construct it when it should be in the changelog.
Do you mean performing multiple parallel page migrations at the same time and show all the page migration time?
I mean that the test case that was used to generate the bandwidth utilisation figures should be included.
Signature · 8 lines
Mel Gorman SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 1 line · Mel Gorman
On Thu, Mar 09, 2017 at 05:46:16PM -0600, Zi Yan wrote:
[...]
Quoted 9 lines
Quoted 4 lines
I understand your concern on CPU utilization impact. I think checking CPU utilization and only using idle CPUs could potentially avoid this problem.
That will be costly to detect actually. It would require poking into the scheduler core and incurring a number of cache misses for a race-prone operation that may not succeed. Even if you do it, it'll still be brought up that the serialised case should be optimised first.
do not forget that seeing idle cpus is not a sufficient criterion to use it for parallel migration. There might be other policies you are not aware of from the MM code to keep them idle (power saving and who knows what else). Developing a reasonable strategy for spreading the load to different CPUs is really hard, much harder than you can imaging I suspect (just look at how hard it was and I long it took to get to a reasonable scheduler driven frequency scaling/power governors).
Signature · 8 lines
Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 28 lines · Mel Gorman
Quoted 10 lines · Anshuman Khandual
Quoted 1 line
Any comments, suggestions are welcome.
Hello Vlastimil/Michal/Minchan/Mel/Dave,
Apart from the comments from Naoya on a different thread posted by Zi Yan, I did not get any more review comments on this series. Could you please kindly have a look on the over all design and its benefits from page migration performance point of view and let me know your views. Thank you.
I didn't look into the patches in detail except to get a general feel for how it works and I'm not convinced that it's a good idea at all.
I accept that memory bandwidth utilisation may be higher as a result but consider the impact. THP migrations are relatively rare and when they occur, it's in the context of a single thread. To parallelise the copy, an allocation, kmap and workqueue invocation are required. There may be a long delay before the workqueue item can start which may exceed the time to do a single copy if the CPUs on a node are saturated. Furthermore, a single thread can preempt operations of other unrelated threads and incur CPU cache pollution and future misses on unrelated CPUs. It's compounded by the fact that a high priority system workqueue is used to do the operation, one that is used for CPU hotplug operations and rolling back when a netdevice fails to be registered. It treats a hugepage copy as an essential operation that can preempt all other work which is very questionable.
Quoted 4 lines
The series leader has no details on a workload that is bottlenecked by THP migrations and even if it is, the primary question should be *why* THP migrations are so frequent and alleviating that instead of preempting multiple CPUs to do the work.
FWIW I very much agree here and the follow up reply. Making migration itself parallel is a hard task. You should start simple and optimize the current code first and each step accompany with numbers. Parallel migration should be the very last step - if it is needed at all of course. I am quite skeptical that a reasonable parallel load balancing is achievable without a large maintenance cost and/or predictable behavior.
Signature · 8 lines
Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>