page migration

The attached tar file contains a version of the mhp3 patch with the following properties:

(1) It splits out the memory migration patches into a separate series file.
(2) The remaining patches are in the hotplug directory with its own series files.
(3) Rollup patches for the two sets of patches are included.

If one applies the memory_migration patches first, the result compiles and links but I admit I have not tested it.

I've been unable to get (either) memory hotplug patch to compile. It won't compile for Altix at all, because Altix requires NUMA. I tried it on a Pentium machine, but apparently I didn't grab the correct config.

Anyway, the fact that the diff shows the split out patches are equivalent to the full mhp3 patch should be good enough.

(The output of the comparison is included as the file reorder.diff).

I'd like to see this order of patches become the new order for the memory hotplug patch. That way, I won't have to pull the migration patches out of the hotplug patch every time a new one comes out (I need the migration code, but not the hotplug code for a project I am working on.)

Do you suppose this can be done???

Quoted 7 lines

I'd like to see this order of patches become the new order for the memory hotplug patch. That way, I won't have to pull the migration patches out of the hotplug patch every time a new one comes out (I need the migration code, but not the hotplug code for a project I am working on.)

Do you suppose this can be done???

Absolutely. I was simply working them in the order that they were implemented. But, if we want the migration stuff merged first, I have absolutely no problem with putting it first in the patch set.

Next time I publish a tree, I'll see what I can do about producing similar rollups to what you have, with migration broken out from hotplug.

Quoted 9 lines · Dave Hansen

Quoted 7 lines

I'd like to see this order of patches become the new order for the memory hotplug patch. That way, I won't have to pull the migration patches out of the hotplug patch every time a new one comes out (I need the migration code, but not the hotplug code for a project I am working on.)

Do you suppose this can be done???

Quoted 8 lines

Absolutely. I was simply working them in the order that they were implemented. But, if we want the migration stuff merged first, I have absolutely no problem with putting it first in the patch set.

Next time I publish a tree, I'll see what I can do about producing similar rollups to what you have, with migration broken out from hotplug.

Cool. Let me know if I can help at all with that.

Once we get that done I'd like to pursure getting the migration patches proposed for -mm and then mainline. Does that make sense?

(perhaps it will make the hotplug patch easier to accept if we can get the memory migration stuff in first).

Of course, the "standalone" memory migration stuff makes most sense on NUMA, and there is some minor interface changes there to support that (i. e. consider:

migrate_onepage(page);

vs

migrate_onepage_node(page, node);

what the latter does is to call alloc_pages_node() instead of page_cache_alloc() to get the new page.)

This is all to support NUMA process and memory migration, where the required function is to move a process >>and<< its memory from one set of nodes to another. (I should have a patch for these initial interface changes later this week.)

But the real question I am wrestling with at the moment is the following:

"Which approach to a NUMA process and memory migration facility would be more likely to get into the mainline kernel:

(1) One based on the existing memory migration patches, or

(2) something simpler just written for the NUMA process and memory migration case."

My preference would be to build on top of the existing code
from the hotplug project. But the key goal here is to get the code into the mainline. I am a little concerned that the hotlug memory migration code will be regarded as too complicated to get in, and I don't want that to hold up the NUMA process and memory migration facility, which is what I am
working on and we (well, SGI) specifically need.

Suggestions?

Quoted 32 lines · Ray Bryant

Quoted 9 lines · Dave Hansen

Quoted 7 lines

I'd like to see this order of patches become the new order for the memory hotplug patch. That way, I won't have to pull the migration patches out of the hotplug patch every time a new one comes out (I need the migration code, but not the hotplug code for a project I am working on.)

Do you suppose this can be done???

Quoted 8 lines

Absolutely. I was simply working them in the order that they were implemented. But, if we want the migration stuff merged first, I have absolutely no problem with putting it first in the patch set.

Next time I publish a tree, I'll see what I can do about producing similar rollups to what you have, with migration broken out from hotplug.

Cool. Let me know if I can help at all with that.

Once we get that done I'd like to pursure getting the migration patches proposed for -mm and then mainline. Does that make sense?

(perhaps it will make the hotplug patch easier to accept if we can get the memory migration stuff in first).

Of course, the "standalone" memory migration stuff makes most sense on NUMA, and there is some minor interface changes there to support that (i. e. consider:

Quoted 8 lines

migrate_onepage(page);

vs

migrate_onepage_node(page, node);

what the latter does is to call alloc_pages_node() instead of page_cache_alloc() to get the new page.)

Memory migration makes sense for defragmentation too.

I think we enough arguments for merging the migration code first, as you suggest.

Its also easier to merge part-by-part than everything in one bunch.

Yes?

Quoted 7 lines · Marcelo Tosatti

Memory migration makes sense for defragmentation too.

I think we enough arguments for merging the migration code first, as you suggest.

Quoted 3 lines

Its also easier to merge part-by-part than everything in one bunch.

Yes?

Absolutely. I guess the only question is when to propose the merge with -mm etc. Is your defragmentation code in a good enough state to be proposed as well, or should we wait a bit?

I think we need at least one user of the code before we can propose that the memory migration code be merged, or do you think we the arguments are strong enough we can proceed with users "pending"?

Quoted 7 lines

Quoted 5 lines · Marcelo Tosatti

Memory migration makes sense for defragmentation too.

I think we enough arguments for merging the migration code first, as you suggest.

Quoted 7 lines

Quoted 3 lines

Its also easier to merge part-by-part than everything in one bunch.

Yes?

Absolutely. I guess the only question is when to propose the merge with -mm etc. Is your defragmentation code in a good enough state to be proposed as well, or should we wait a bit?

No, we have to wait - its not ready yet.

But it is really simple and small, as soon as the "asynchronous" memory migration is working.

Quoted 3 lines

I think we need at least one user of the code before we can propose that the memory migration code be merged, or do you think we the arguments are strong enough we can proceed with users "pending"?

IMO the arguments are strong enough that we can proceed with the current state. I'm all for it.

Andrew knows the importance and the users of the memory migration infrastructure.

Dave, Hirokazu, what are your thoughts on this

Shall we CC Andrew?

Quoted 17 lines

Quoted 4 lines

Absolutely. I guess the only question is when to propose the merge with -mm etc. Is your defragmentation code in a good enough state to be proposed as well, or should we wait a bit?

No, we have to wait - its not ready yet.

But it is really simple and small, as soon as the "asynchronous" memory migration is working.

Quoted 3 lines

I think we need at least one user of the code before we can propose that the memory migration code be merged, or do you think we the arguments are strong enough we can proceed with users "pending"?

IMO the arguments are strong enough that we can proceed with the current state. I'm all for it.

Quoted 3 lines

Andrew knows the importance and the users of the memory migration infrastructure.

Dave, Hirokazu, what are your thoughts on this

Andrew is interested in our approach.
With Ray's help, it will proceed faster and become stable soon:)

Quoted 2 lines

Shall we CC Andrew?

Quoted 23 lines

Quoted 23 lines

Quoted 6 lines

Absolutely. I guess the only question is when to propose the merge with -mm
etc. Is your defragmentation code in a good enough state to be proposed as
well, or should we wait a bit?

No, we have to wait - its not ready yet.

But it is really simple and small, as soon as the "asynchronous" memory migration is working.

Quoted 5 lines

I think we need at least one user of the code before we can propose that the
memory migration code be merged, or do you think we the arguments are strong
enough we can proceed with users "pending"?

IMO the arguments are strong enough that we can proceed with the current state.
I'm all for it.

Quoted 14 lines

Quoted 4 lines

Andrew knows the importance and the users of the memory migration infrastructure.

Dave, Hirokazu, what are your thoughts on this

Andrew is interested in our approach.
With Ray's help, it will proceed faster and become stable soon:)

Quoted 2 lines

Shall we CC Andrew?

If it is ok with everyone, I will email Andrew and see how he'd like to proceed on this, whether he'd prefer we contribute a solid "user" of the page migration code with a merged page migration patch, or if it would be ok to
submit the page migration code stand alone, given that there are multiple users "pending".

Of course, I come to this effort late in the game, and if anyone else would prefer to do that instead, I will happily oblige them.

Hi Ray!

IMO, if you are in a hurry to get SGI's NUMA process/migration facility in mainline, we will have duplicated efforts/code in the future.

My suggestion is to wait for the memory migration code to get in. It might take sometime, yes, which
is a good thing from stabilization/maturity POV. If SGI has deadlines for the process/migration facility then implement your own (can even be based on the current migration infrastructure) and do no try to merge it to the tree, please.

This is also valid for the memory defragmentation effort I'm working on, I could have implemented standalone code myself (and I actually did), but it was a duplication, to some extent, to what was already being done at the migration code.

So I had to try to change the migration code to attend my needs, and in the process me and Hirokazu came up with the migration cache, which is an advantage for every user of the memory migration.

Unification - you get the point.

Once we get that done I'd like to pursure getting the migration patches proposed for -mm and then mainline. Does that make sense?

(perhaps it will make the hotplug patch easier to accept if we can get the memory migration stuff in first).

Of course, the "standalone" memory migration stuff makes most sense on NUMA, and there is some minor interface changes there to support that (i. e. consider:

migrate_onepage(page);

vs

migrate_onepage_node(page, node);

what the latter does is to call alloc_pages_node() instead of page_cache_alloc() to get the new page.)

This is all to support NUMA process and memory migration, where the required function is to move a process >>and<< its memory from one set of nodes to another. (I should have a patch for these initial interface changes later this week.)

But the real question I am wrestling with at the moment is the following:

"Which approach to a NUMA process and memory migration facility would be more likely to get into the mainline kernel:

(1) One based on the existing memory migration patches, or

(2) something simpler just written for the NUMA process and memory migration case."

My preference would be to build on top of the existing code
from the hotplug project. But the key goal here is to get the code into the mainline. I am a little concerned that the hotlug memory migration code will be regarded as too complicated to get in, and I don't want that to hold up the NUMA process and memory migration facility, which is what I am working on and we (well, SGI) specifically need.

I was going to do hotplug first only because it created a requirement for migration. Since you have a separate requirement for it, I have no objection to doing migration first.

Quoted 7 lines

Of course, the "standalone" memory migration stuff makes most sense on NUMA, and there is some minor interface changes there to support that (i. e. consider:

migrate_onepage(page);

vs

Quoted 4 lines

migrate_onepage_node(page, node);

what the latter does is to call alloc_pages_node() instead of page_cache_alloc() to get the new page.)

We might as well just change all of the users over to the NUMA version from the start. Having 2 different functions just causes confusion.

Quoted 5 lines

This is all to support NUMA process and memory migration, where the required function is to move a process >>and<< its memory from one set of nodes to another. (I should have a patch for these initial interface changes later this week.)

Well, moving the process itself isn't very hard, right? That's just a schedule().

Anyway, I'd love to see a simpler version if it's possible. I'd just keep Marcello and Hirokazu in the loop if you're going to try.

-- Dave

If the consensus is that the correct way to go is to propose the memory migration patches as they are, then that is fine by me. I will get my "NUMA process and memory migration" patch working on top of that (so that we have a user) and then work with Andrew to get them into -mm and then see what happens from there.

Please don't limit the scope of page migration to that; cross-zone page migration is needed to resolve pathologies arising in swapless systems.

-- wli

Quoted 7 lines

Quoted 7 lines

Of course, the "standalone" memory migration stuff makes most sense on NUMA, and there is some minor interface changes there to support that (i. e. consider:

migrate_onepage(page);

vs

Quoted 9 lines

Quoted 4 lines

migrate_onepage_node(page, node);

what the latter does is to call alloc_pages_node() instead of page_cache_alloc() to get the new page.)

We might as well just change all of the users over to the NUMA version from the start. Having 2 different functions just causes confusion.

Yes, especially since alloc_pages_node() is defined regardless of whether NUMA is defined (I've found out by some code inspection). So in the non-DISCONTIGMEM cases, the node argument would just be ignored. I'll put together a patch that moves the interface over to

migrate_onepage(page, node)

and fixes up the callers in the memory hotplug patches.

I also think we should rewrite page allocation in the memory migration code, as the latest -mm tree includes NUMA aware page allocator. I guess you should also care about mm/mempolicy.c and expand it for your purpose. If memory migration is called after moving a process, a new page would be allocated form a proper node automatically.

Have you checked mm/mempolicy.c?

Thanks,
Hirokazu Takahashi.

My thinking on this was to update the mempolicy after page migration. This works for my purposes since my plan is to

(1)  suspend the process via SIGSTOP
(2)  update the mempolicy
(3)  migrate the process's pages
(4)  migrate the process to the new cpu via set_schedaffinity()
(5)  resume the process via SIGCONT

These steps are to be performed via a user space program that implements the actual migration function; the (2)-(4) are just the system calls that implement this. This keeps some of the function (i. e. which processes to migrate) out of the kernel and allows the user some flexibility in what order operations are performed as well as other functions that may go along with this migration request. (The actual function we are trying to implement is to support >>job<< migration from one set of NUMA nodes to another, and a job may consist of several processes.)

On Tue, 2005-01-04 at 11:30 -0600, Ray Bryant wrote:

We already have scheduler code which has some knowledge of when a process is dragged from one node to another. Combined with the per-node RSS, could we make a decision about when a process needs to have migration performed on its pages on a more automatic basis, without the syscalls?

We could have a tunable for how aggressive this mechanism is, so that the process wouldn't start running again on the more strict SGI machines until a very large number of the pages are pulled over. However, on machines where process latency is more of an issue, the tunable could be set to a much less aggressive value.

This would give normal, somewhat less exotic, NUMA machines the benefits of page migration without the need for the process owner to do anything manually to them, while also making sure that we keep the number of interfaces to the migration code to a relative minimum.

-- Dave

Attached is a trivial little patch that fixes the names on the initial #ifdef and #define in linux/include/mmigrate.h to match that file's name (it appears this was copied over from some memory hotplug patch and was never updated....)

Marcello,

Attached is a patch which fixes some compiler warnings in mmigrate.c that I was getting with the migration cache code. The only substantive change was to change:

        /* Wait for all operations against the page to finish. */
        ret = migrate_fn(page, newpage, &vlist);
        switch (ret) {
        default:
                /* The page is busy. Try it later. */
                goto out_busy;
        case -ENOENT:
                /* The file the page belongs to has been truncated. */
                page_cache_get(page);
                page_cache_release(newpage);
                newpage->mapping = NULL;
                /* fall thru */
        case 0:
                /* fall thru */
        }

in generic_migrate_page(), to:

        /* Wait for all operations against the page to finish. */
        ret = migrate_fn(page, newpage, &vlist);
        switch (ret) {
        case -ENOENT:
                /* The file the page belongs to has been truncated. */
                page_cache_get(page);
                page_cache_release(newpage);
                newpage->mapping = NULL;
                /* fall thru */
        case 0:
                break;
        default:
                /* The page is busy. Try it later. */
                goto out_busy;
        }

This change was made to get rid of the warning:

mm/mmigrate.c:500: warning: deprecated use of label at end of compound statement

I suppose you used the previous order to eliminate an extra branch or some such. Do you have any other suggestion on how to eliminate that warning?

Signature · 12 lines

Best Regards,
Ray
-----------------------------------------------
                  Ray Bryant
512-453-9679 (work)         512-507-7807 (cell)
[email protected]             [email protected]
The box said: "Requires Windows 98 or better",
           so I installed Linux.
-----------------------------------------------

Index: linux-2.6.10-rc2-mm4-page-migration-only/include/linux/swap.h
===================================================================

include/linux/swap.h +1 -1

@@ -258,7 +258,7 @@ static inline int remove_exclusive_swap_
 {
 	return __remove_exclusive_swap_page(p, 0);
 }
-extern int migration_remove_entry(swp_entry_t);
+extern void migration_remove_entry(swp_entry_t);
 struct backing_dev_info;
 
 extern struct swap_list_t swap_list;

Index: linux-2.6.10-rc2-mm4-page-migration-only/mm/mmigrate.c
===================================================================

mm/mmigrate.c +24 -27

@@ -79,7 +79,6 @@ struct page *lookup_migration_cache(int 
 
 void migration_duplicate(swp_entry_t entry)
 {
-	int offset;
 	struct counter *cnt;
 
 	read_lock_irq(&migration_space.tree_lock);

@@ -96,32 +95,11 @@ void remove_from_migration_cache(struct 
         idr_remove(&migration_idr, id);
 	radix_tree_delete(&migration_space.page_tree, id);
 	ClearPageSwapCache(page);
-	page->private = NULL;
+	page->private = 0;
 	write_unlock_irq(&migration_space.tree_lock);
 }
 
-// FIXME: if the page is locked will it be correctly removed from migr cache?
-// check races
-
-int migration_remove_entry(swp_entry_t entry)
-{
-	struct page *page;
-	
-	page = find_get_page(&migration_space, entry.val);
-
-	if (!page)
-		BUG();
-
-	lock_page(page);	
-
-	migration_remove_reference(page, 1);
-
-	unlock_page(page);
-
-	page_cache_release(page);
-}
-
-int migration_remove_reference(struct page *page, int dec)
+void migration_remove_reference(struct page *page, int dec)
 {
 	struct counter *c;
 	swp_entry_t entry;

@@ -145,6 +123,28 @@ int migration_remove_reference(struct pa
 	}
 }
 
+
+// FIXME: if the page is locked will it be correctly removed from migr cache?
+// check races
+
+void migration_remove_entry(swp_entry_t entry)
+{
+	struct page *page;
+	
+	page = find_get_page(&migration_space, entry.val);
+
+	if (!page)
+		BUG();
+
+	lock_page(page);	
+
+	migration_remove_reference(page, 1);
+
+	unlock_page(page);
+
+	page_cache_release(page);
+}
+
 int detach_from_migration_cache(struct page *page)
 {
 	lock_page(page);	

@@ -486,9 +486,6 @@ generic_migrate_page(struct page *page, 
 	/* Wait for all operations against the page to finish. */
 	ret = migrate_fn(page, newpage, &vlist);
 	switch (ret) {
-	default:
-		/* The page is busy. Try it later. */
-		goto out_busy;
 	case -ENOENT:
 		/* The file the page belongs to has been truncated. */
 		page_cache_get(page);

@@ -496,7 +493,10 @@ generic_migrate_page(struct page *page, 
 		newpage->mapping = NULL;

Quoted 3 lines

I've been unable to get (either) memory hotplug patch to compile. It won't compile for Altix at all, because Altix requires NUMA. I tried it on a Pentium machine, but apparently I didn't grab the correct config.

Hmmm. Did you check the configs here?

	http://sr71.net/patches/2.6.10/2.6.10-rc2-mm4-mhp3/configs/

Hello Ray-san.

Quoted 7 lines

Quoted 3 lines

I've been unable to get (either) memory hotplug patch to compile. It won't compile for Altix at all, because Altix requires NUMA. I tried it on a Pentium machine, but apparently I didn't grab the correct config.

Hmmm. Did you check the configs here?

	http://sr71.net/patches/2.6.10/2.6.10-rc2-mm4-mhp3/configs/

CONFIG_NUMA with memory hotplug is disabled on -mhp3, because some functions of memory hotplug are not defined yet and some works like pgdat allocation are necessary.
.
I posted patches for them before holidays to LHMS.
http://sourceforge.net/mailarchive/forum.php?forum_id=223&max_rows=25&style=ultimate&viewmonth=200412

It is still for IA32. But, I would like to start works for IA64. I guess it won't be duplication against your works. But If you find something wrong, please let me know.

Bye.