Skip to contentSkip to conversation

do_migrate_range: avoid failure as much as possible

PATCH In review Bob Liu 3 participants 15 messages Oct 25, 2010

Subsystems (2)
lore
Share & tools
lore

Bob Liu

author
#0 lore raw

PATCH 1/1 do_migrate_range: avoid failure as much as possible

It's normal for isolate_lru_page() to fail at times. The failures are typically temporal and may well go away when offline_pages() retries the call. So it seems more reasonable to migrate as much as possible to increase the chance of complete success in next retry.

This patch remove page_count() check and remove putback_lru_pages() and call migrate_pages() regardless of not_managed to reduce failure as much as possible.

---
 mm/memory_hotplug.c |   12 ------------
 1 files changed, 0 insertions(+), 12 deletions(-)
mm/memory_hotplug.c +0 -12
index a4cfcdc..b64cc9b 100644
@@ -687,7 +687,6 @@ do_migrate_range(unsigned long start_pfn, unsigned long end_pfn)
 	unsigned long pfn;
 	struct page *page;
 	int move_pages = NR_OFFLINE_AT_ONCE_PAGES;
-	int not_managed = 0;
 	int ret = 0;
 	LIST_HEAD(source);
 
@@ -709,10 +708,6 @@ do_migrate_range(unsigned long start_pfn, unsigned long end_pfn)
 					    page_is_file_cache(page));
 
 		} else {
-			/* Becasue we don't have big zone->lock. we should
-			   check this again here. */
-			if (page_count(page))
-				not_managed++;
 #ifdef CONFIG_DEBUG_VM
 			printk(KERN_ALERT "removing pfn %lx from LRU failed\n",
 			       pfn);
@@ -720,13 +715,6 @@ do_migrate_range(unsigned long start_pfn, unsigned long end_pfn)
 #endif
 		}
 	}
-	ret = -EBUSY;
-	if (not_managed) {
-		if (!list_empty(&source))
-			putback_lru_pages(&source);
-		goto out;
-	}
-	ret = 0;
 	if (list_empty(&source))
 		goto out;
 	/* this function returns # of failed pages */
Signature · 7 lines
1.6.3.3

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#1 lore raw
Quoted 10 lines · On Mon, 25 Oct 2010 10:47:31 +0800 Bob Liu <[email protected]> wrote:

It's normal for isolate_lru_page() to fail at times. The failures are typically temporal and may well go away when offline_pages() retries the call. So it seems more reasonable to migrate as much as possible to increase the chance of complete success in next retry.

This patch remove page_count() check and remove putback_lru_pages() and call migrate_pages() regardless of not_managed to reduce failure as much as possible.

-EBUSY should be returned.

-Kame

Quoted 45 lines
---
 mm/memory_hotplug.c |   12 ------------
 1 files changed, 0 insertions(+), 12 deletions(-)
mm/memory_hotplug.c +0 -12
index a4cfcdc..b64cc9b 100644
@@ -687,7 +687,6 @@ do_migrate_range(unsigned long start_pfn, unsigned long end_pfn)
 	unsigned long pfn;
 	struct page *page;
 	int move_pages = NR_OFFLINE_AT_ONCE_PAGES;
-	int not_managed = 0;
 	int ret = 0;
 	LIST_HEAD(source);
 
@@ -709,10 +708,6 @@ do_migrate_range(unsigned long start_pfn, unsigned long end_pfn)
 					    page_is_file_cache(page));
 
 		} else {
-			/* Becasue we don't have big zone->lock. we should
-			   check this again here. */
-			if (page_count(page))
-				not_managed++;
 #ifdef CONFIG_DEBUG_VM
 			printk(KERN_ALERT "removing pfn %lx from LRU failed\n",
 			       pfn);
@@ -720,13 +715,6 @@ do_migrate_range(unsigned long start_pfn, unsigned long end_pfn)
 #endif
 		}
 	}
-	ret = -EBUSY;
-	if (not_managed) {
-		if (!list_empty(&source))
-			putback_lru_pages(&source);
-		goto out;
-	}
-	ret = 0;
 	if (list_empty(&source))
 		goto out;
 	/* this function returns # of failed pages */
Signature · 1 line
1.6.3.3

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#2 lore raw
Quoted 8 lines · KAMEZAWA Hiroyuki
Quoted 5 lines · On Mon, 25 Oct 2010 10:47:31 +0800 Bob Liu <[email protected]> wrote:

It's normal for isolate_lru_page() to fail at times. The failures are typically temporal and may well go away when offline_pages() retries the call. So it seems more reasonable to migrate as much as possible to increase the chance of complete success in next retry.

Quoted 7 lines
Quoted 5 lines

This patch remove page_count() check and remove putback_lru_pages() and call migrate_pages() regardless of not_managed to reduce failure as much as possible.

-EBUSY should be returned.

It does return -EBUSY when ALL pages cannot be isolated from LRU (or is non-LRU pages at all). That means offline_pages() will repeat calls to do_migrate_range() as fast as possible as long as it can make progress.

Is that behavior good enough? It does need some comment for this non-obvious return value.

btw, the caller side code can be simplified (no behavior change).

mm/memory_hotplug.c +5 -9
index dd186c1..606d358 100644
@@ -848,17 +848,13 @@ repeat:
 	pfn = scan_lru_pages(start_pfn, end_pfn);
 	if (pfn) { /* We have page on LRU */
 		ret = do_migrate_range(pfn, end_pfn);
-		if (!ret) {
-			drain = 1;
-			goto repeat;
-		} else {
-			if (ret < 0)
-				if (--retry_max == 0)
-					goto failed_removal;
+		if (ret < 0) {
+			if (--retry_max <= 0)
+				goto failed_removal;
 			yield();
-			drain = 1;
-			goto repeat;
 		}
+		drain = 1;
+		goto repeat;
 	}
 	/* drain all zone's lru pagevec, this is asyncronous... */
 	lru_add_drain_all();

Thanks,
Fengguang

Quoted 37 lines
Quoted 37 lines
---
 mm/memory_hotplug.c |   12 ------------
 1 files changed, 0 insertions(+), 12 deletions(-)
mm/memory_hotplug.c +0 -11
index a4cfcdc..b64cc9b 100644
@@ -687,7 +687,6 @@ do_migrate_range(unsigned long start_pfn, unsigned long end_pfn)
 	unsigned long pfn;
 	struct page *page;
 	int move_pages = NR_OFFLINE_AT_ONCE_PAGES;
-	int not_managed = 0;
 	int ret = 0;
 	LIST_HEAD(source);
 
@@ -709,10 +708,6 @@ do_migrate_range(unsigned long start_pfn, unsigned long end_pfn)
 					    page_is_file_cache(page));
 
 		} else {
-			/* Becasue we don't have big zone->lock. we should
-			   check this again here. */
-			if (page_count(page))
-				not_managed++;
 #ifdef CONFIG_DEBUG_VM
 			printk(KERN_ALERT "removing pfn %lx from LRU failed\n",
 			       pfn);
@@ -720,13 +715,6 @@ do_migrate_range(unsigned long start_pfn, unsigned long end_pfn)
 #endif
 		}
 	}
-	ret = -EBUSY;
-	if (not_managed) {
-		if (!list_empty(&source))
-			putback_lru_pages(&source);
-		goto out;
-	}
Quoted 8 lines
Quoted 8 lines
mm/memory_hotplug.c +0 -1
-	ret = 0;
 	if (list_empty(&source))
 		goto out;
 	/* this function returns # of failed pages */
Signature · 1 line
1.6.3.3

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#3 lore raw
Quoted 44 lines · Wu Fengguang
Quoted 8 lines · KAMEZAWA Hiroyuki
Quoted 5 lines · On Mon, 25 Oct 2010 10:47:31 +0800 Bob Liu <[email protected]> wrote:

It's normal for isolate_lru_page() to fail at times. The failures are typically temporal and may well go away when offline_pages() retries the call. So it seems more reasonable to migrate as much as possible to increase the chance of complete success in next retry.

Quoted 7 lines
Quoted 5 lines

This patch remove page_count() check and remove putback_lru_pages() and call migrate_pages() regardless of not_managed to reduce failure as much as possible.

-EBUSY should be returned.

It does return -EBUSY when ALL pages cannot be isolated from LRU (or is non-LRU pages at all). That means offline_pages() will repeat calls to do_migrate_range() as fast as possible as long as it can make progress.

Is that behavior good enough? It does need some comment for this non-obvious return value.

btw, the caller side code can be simplified (no behavior change).

mm/memory_hotplug.c +2 -7
index dd186c1..606d358 100644
@@ -848,17 +848,13 @@ repeat:
 	pfn = scan_lru_pages(start_pfn, end_pfn);
 	if (pfn) { /* We have page on LRU */
 		ret = do_migrate_range(pfn, end_pfn);
-		if (!ret) {
-			drain = 1;
-			goto repeat;
-		} else {
-			if (ret < 0)
-				if (--retry_max == 0)
-					goto failed_removal;
+		if (ret < 0) {
+			if (--retry_max <= 0)

mm/memory_hotplug.c repeat: · line 860

Quoted 10 lines
mm/memory_hotplug.c +3 -2
+				goto failed_removal;
 			yield();
-			drain = 1;
-			goto repeat;
 		}
+		drain = 1;
+		goto repeat;
 	}
 	/* drain all zone's lru pagevec, this is asyncronous... */
 	lru_add_drain_all();

And it seems the costly drain operations could be avoided as long as it's making progress. What do you think?

mm/memory_hotplug.c +1 -1
@@ -852,8 +852,8 @@ repeat:
 			if (--retry_max <= 0)
 				goto failed_removal;
 			yield();
+			drain = 1;
 		}
-		drain = 1;
 		goto repeat;
 	}
 	/* drain all zone's lru pagevec, this is asyncronous... */

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#4 lore raw
Quoted 62 lines · Wu Fengguang
Quoted 44 lines · Wu Fengguang
Quoted 8 lines · KAMEZAWA Hiroyuki
Quoted 5 lines · On Mon, 25 Oct 2010 10:47:31 +0800 Bob Liu <[email protected]> wrote:

It's normal for isolate_lru_page() to fail at times. The failures are typically temporal and may well go away when offline_pages() retries the call. So it seems more reasonable to migrate as much as possible to increase the chance of complete success in next retry.

Quoted 7 lines
Quoted 5 lines

This patch remove page_count() check and remove putback_lru_pages() and call migrate_pages() regardless of not_managed to reduce failure as much as possible.

-EBUSY should be returned.

It does return -EBUSY when ALL pages cannot be isolated from LRU (or is non-LRU pages at all). That means offline_pages() will repeat calls to do_migrate_range() as fast as possible as long as it can make progress.

Is that behavior good enough? It does need some comment for this non-obvious return value.

btw, the caller side code can be simplified (no behavior change).

mm/memory_hotplug.c +2 -7
index dd186c1..606d358 100644
@@ -848,17 +848,13 @@ repeat:
 	pfn = scan_lru_pages(start_pfn, end_pfn);
 	if (pfn) { /* We have page on LRU */
 		ret = do_migrate_range(pfn, end_pfn);
-		if (!ret) {
-			drain = 1;
-			goto repeat;
-		} else {
-			if (ret < 0)
-				if (--retry_max == 0)
-					goto failed_removal;
+		if (ret < 0) {
+			if (--retry_max <= 0)

mm/memory_hotplug.c repeat: · line 860

Quoted 10 lines
mm/memory_hotplug.c +3 -2
+				goto failed_removal;
 			yield();
-			drain = 1;
-			goto repeat;
 		}
+		drain = 1;
+		goto repeat;
 	}
 	/* drain all zone's lru pagevec, this is asyncronous... */
 	lru_add_drain_all();

And it seems the costly drain operations could be avoided as long as it's making progress. What do you think?

mm/memory_hotplug.c repeat: · line 859

Quoted 9 lines
mm/memory_hotplug.c +1 -1
@@ -852,8 +852,8 @@ repeat:
 			if (--retry_max <= 0)
 				goto failed_removal;
 			yield();
+			drain = 1;
 		}
-		drain = 1;
 		goto repeat;
 	}
 	/* drain all zone's lru pagevec, this is asyncronous... */

This is a more heavy weight patch for the above one-liner change. I don't have real experiences to understand the requirements for memory hot remove, so the idea may be way too imaginary.

mm/memory_hotplug.c +9 -13
@@ -788,7 +788,7 @@ static int offline_pages(unsigned long s
 {
 	unsigned long pfn, nr_pages, expire;
 	long offlined_pages;
-	int ret, drain, retry_max, node;
+	int ret, retry_max, node;
 	struct zone *zone;
 	struct memory_notify arg;
 
@@ -827,7 +827,6 @@ static int offline_pages(unsigned long s
 
 	pfn = start_pfn;
 	expire = jiffies + timeout;
-	drain = 0;
 	retry_max = 5;
 repeat:
 	/* start memory hot removal */
@@ -838,13 +837,6 @@ repeat:
 	if (signal_pending(current))
 		goto failed_removal;
 	ret = 0;
-	if (drain) {
-		lru_add_drain_all();
-		flush_scheduled_work();
-		cond_resched();
-		drain_all_pages();
-	}
-
 	pfn = scan_lru_pages(start_pfn, end_pfn);
 	if (pfn) { /* We have page on LRU */
 		ret = do_migrate_range(pfn, end_pfn);
@@ -852,15 +844,19 @@ repeat:
 			if (--retry_max <= 0)
 				goto failed_removal;
 			yield();
+			lru_add_drain_all();
+			flush_scheduled_work();
+			cond_resched();
+			drain_all_pages();
 		}
-		drain = 1;
 		goto repeat;
 	}
-	/* drain all zone's lru pagevec, this is asyncronous... */
+
+	/* drain all zone's lru pagevec, this is asynchronous... */
 	lru_add_drain_all();
 	flush_scheduled_work();
 	yield();
-	/* drain pcp pages , this is synchrouns. */
+	/* drain pcp pages , this is asynchronous. */
 	drain_all_pages();
 	/* check again */
 	offlined_pages = check_pages_isolated(start_pfn, end_pfn);
@@ -869,7 +865,7 @@ repeat:
 		goto failed_removal;
 	}
 	printk(KERN_INFO "Offlined Pages %ld\n", offlined_pages);
-	/* Ok, all of our target is islaoted.
+	/* Ok, all of our target is isolated.
 	   We cannot do rollback at this point. */
 	offline_isolated_pages(start_pfn, end_pfn);
 	/* reset pagetype flags and makes migrate type to be MOVABLE */

Thanks,
Fengguang

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#5 lore raw
In reply to Wu Fengguang
Quoted 13 lines · On Mon, 25 Oct 2010 10:57:03 +0800 Wu Fengguang <[email protected]> wrote:
Quoted 12 lines · KAMEZAWA Hiroyuki
Quoted 9 lines · On Mon, 25 Oct 2010 10:47:31 +0800 Bob Liu <[email protected]> wrote:

It's normal for isolate_lru_page() to fail at times. The failures are typically temporal and may well go away when offline_pages() retries the call. So it seems more reasonable to migrate as much as possible to increase the chance of complete success in next retry.

This patch remove page_count() check and remove putback_lru_pages() and call migrate_pages() regardless of not_managed to reduce failure as much as possible.

Quoted 9 lines
Quoted 3 lines
Quoted 1 line

-EBUSY should be returned.

It does return -EBUSY when ALL pages cannot be isolated from LRU (or is non-LRU pages at all). That means offline_pages() will repeat calls to do_migrate_range() as fast as possible as long as it can make progress.

I read the patch wrong ? "ret = -EBUSY" is dropped and "ret" will be 0 or just a return code of migrate_page().

Quoted 20 lines

Is that behavior good enough? It does need some comment for this non-obvious return value.

btw, the caller side code can be simplified (no behavior change).

mm/memory_hotplug.c +0 -7
index dd186c1..606d358 100644
@@ -848,17 +848,13 @@ repeat:
 	pfn = scan_lru_pages(start_pfn, end_pfn);
 	if (pfn) { /* We have page on LRU */
 		ret = do_migrate_range(pfn, end_pfn);
-		if (!ret) {
-			drain = 1;
-			goto repeat;
-		} else {
-			if (ret < 0)
-				if (--retry_max == 0)
-					goto failed_removal;

mm/memory_hotplug.c repeat: · line 858

Quoted 10 lines
mm/memory_hotplug.c +5 -2
+		if (ret < 0) {
+			if (--retry_max <= 0)
+				goto failed_removal;
 			yield();
-			drain = 1;
-			goto repeat;
 		}
+		drain = 1;
+		goto repeat;
 	}

This changes behavior.

This "ret" can be > 0 because migrate_page()'s return code is "Return: Number of pages not migrated or error code."

Then,
ret < 0 ===> maybe ebusy
ret > 0 ===> some pages are not migrated. maybe PG_writeback or some ret == 0 ===> ok, all condition green. try next chunk soon.

Then, I added "yield()" and --retrym_max for !ret cases.

Thanks,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#6 lore raw
Quoted 11 lines · On Mon, 25 Oct 2010 12:05:50 +0900 KAMEZAWA Hiroyuki…

This changes behavior.

This "ret" can be > 0 because migrate_page()'s return code is "Return: Number of pages not migrated or error code."

Then,
ret < 0 ===> maybe ebusy
ret > 0 ===> some pages are not migrated. maybe PG_writeback or some ret == 0 ===> ok, all condition green. try next chunk soon.

Then, I added "yield()" and --retrym_max for !ret cases.

                                               ^^^^^^^^
						wrong.

The code here does

ret == 0 ==> ok, all condition green, try next chunk.
ret > 0 ==> all pages are isolated but some pages cannot be migrated. maybe under I/O
do yield.
ret < 0 ==> some pages may not be able to be isolated. reduce retrycount and yield()

Thanks,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#7 lore raw
Quoted 13 lines · KAMEZAWA Hiroyuki
Quoted 10 lines · On Mon, 25 Oct 2010 12:05:50 +0900 KAMEZAWA Hiroyuki…

This changes behavior.

This "ret" can be > 0 because migrate_page()'s return code is "Return: Number of pages not migrated or error code."

Then,
ret < 0 ===> maybe ebusy
ret > 0 ===> some pages are not migrated. maybe PG_writeback or some ret == 0 ===> ok, all condition green. try next chunk soon.

Quoted 7 lines
Quoted 1 line

Then, I added "yield()" and --retrym_max for !ret cases.

                                               ^^^^^^^^
						wrong.

The code here does

ret == 0 ==> ok, all condition green, try next chunk.

It seems reasonable to remove the drain operations for "ret == 0" case. That would help large NUMA boxes noticeably I guess.

Quoted 2 lines

ret > 0 ==> all pages are isolated but some pages cannot be migrated. maybe under I/O
do yield.

Don't know how to deal with the possible "migration fail" pages -- sorry I have no idea about that situation at all.

Perhaps, OOM while offlining pages?

Quoted 1 line

ret < 0 ==> some pages may not be able to be isolated. reduce retrycount and yield()

Makes good sense.

Thanks,
Fengguang

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#8 lore raw
Quoted 18 lines · On Mon, 25 Oct 2010 11:48:33 +0800 Wu Fengguang <[email protected]> wrote:
Quoted 17 lines · KAMEZAWA Hiroyuki
Quoted 11 lines · On Mon, 25 Oct 2010 12:05:50 +0900 KAMEZAWA Hiroyuki…

This changes behavior.

This "ret" can be > 0 because migrate_page()'s return code is "Return: Number of pages not migrated or error code."

Then,
ret < 0 ===> maybe ebusy
ret > 0 ===> some pages are not migrated. maybe PG_writeback or some ret == 0 ===> ok, all condition green. try next chunk soon.

Then, I added "yield()" and --retrym_max for !ret cases.

                                               ^^^^^^^^
						wrong.
Quoted 7 lines
Quoted 3 lines

The code here does

ret == 0 ==> ok, all condition green, try next chunk.

It seems reasonable to remove the drain operations for "ret == 0" case. That would help large NUMA boxes noticeably I guess.

Maybe.

Quoted 6 lines
Quoted 2 lines

ret > 0 ==> all pages are isolated but some pages cannot be migrated. maybe under I/O
do yield.

Don't know how to deal with the possible "migration fail" pages -- sorry I have no idea about that situation at all.

In typical case, page_count() > 0 by get_user_pages() or PG_writeback is set. All we can do is just waiting.

Quoted 2 lines

Perhaps, OOM while offlining pages?

I never see that..because memory offline is scheduled to be done only when there are free memory.

Thanks,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#9 lore raw
Quoted 33 lines · KAMEZAWA Hiroyuki
Quoted 18 lines · On Mon, 25 Oct 2010 11:48:33 +0800 Wu Fengguang <[email protected]> wrote:
Quoted 17 lines · KAMEZAWA Hiroyuki
Quoted 11 lines · On Mon, 25 Oct 2010 12:05:50 +0900 KAMEZAWA Hiroyuki…

This changes behavior.

This "ret" can be > 0 because migrate_page()'s return code is "Return: Number of pages not migrated or error code."

Then,
ret < 0 ===> maybe ebusy
ret > 0 ===> some pages are not migrated. maybe PG_writeback or some ret == 0 ===> ok, all condition green. try next chunk soon.

Then, I added "yield()" and --retrym_max for !ret cases.

                                               ^^^^^^^^
						wrong.
Quoted 7 lines
Quoted 3 lines

The code here does

ret == 0 ==> ok, all condition green, try next chunk.

It seems reasonable to remove the drain operations for "ret == 0" case. That would help large NUMA boxes noticeably I guess.

Maybe.

Quoted 3 lines
Quoted 2 lines

ret > 0 ==> all pages are isolated but some pages cannot be migrated. maybe under I/O
do yield.

Quoted 6 lines
Quoted 3 lines

Don't know how to deal with the possible "migration fail" pages -- sorry I have no idea about that situation at all.

In typical case, page_count() > 0 by get_user_pages() or PG_writeback is set. All we can do is just waiting.

OK.

Quoted 5 lines
Quoted 2 lines

Perhaps, OOM while offlining pages?

I never see that..because memory offline is scheduled to be done only when there are free memory.

OK.

On OOM migrate_page() will return -ENOMEM, which will be handled in the "ret < 0" case. So it will give up after some retries.

migrate_page() has a comment /* Permanent failure */ when returning positive ret. So it looks safer not to retry indefinitely on the "ret > 0" case?

Then it's reduced to two cases: "ret != 0, cannot make smooth progress, unconditional retries may livelock" and "ret ==0, makes some progress, safe to retry".

Thanks,
Fengguang

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#10 lore raw
Quoted 54 lines · On Mon, 25 Oct 2010 12:06:04 +0800 Wu Fengguang <[email protected]> wrote:
Quoted 33 lines · KAMEZAWA Hiroyuki
Quoted 18 lines · On Mon, 25 Oct 2010 11:48:33 +0800 Wu Fengguang <[email protected]> wrote:
Quoted 17 lines · KAMEZAWA Hiroyuki
Quoted 11 lines · On Mon, 25 Oct 2010 12:05:50 +0900 KAMEZAWA Hiroyuki…

This changes behavior.

This "ret" can be > 0 because migrate_page()'s return code is "Return: Number of pages not migrated or error code."

Then,
ret < 0 ===> maybe ebusy
ret > 0 ===> some pages are not migrated. maybe PG_writeback or some ret == 0 ===> ok, all condition green. try next chunk soon.

Then, I added "yield()" and --retrym_max for !ret cases.

                                               ^^^^^^^^
						wrong.
Quoted 7 lines
Quoted 3 lines

The code here does

ret == 0 ==> ok, all condition green, try next chunk.

It seems reasonable to remove the drain operations for "ret == 0" case. That would help large NUMA boxes noticeably I guess.

Maybe.

Quoted 3 lines
Quoted 2 lines

ret > 0 ==> all pages are isolated but some pages cannot be migrated. maybe under I/O
do yield.

Quoted 6 lines
Quoted 3 lines

Don't know how to deal with the possible "migration fail" pages -- sorry I have no idea about that situation at all.

In typical case, page_count() > 0 by get_user_pages() or PG_writeback is set. All we can do is just waiting.

OK.

Quoted 5 lines
Quoted 2 lines

Perhaps, OOM while offlining pages?

I never see that..because memory offline is scheduled to be done only when there are free memory.

OK.

On OOM migrate_page() will return -ENOMEM, which will be handled in the "ret < 0" case. So it will give up after some retries.

Quoted 8 lines

migrate_page() has a comment /* Permanent failure */ when returning positive ret. So it looks safer not to retry indefinitely on the "ret > 0" case?

Then it's reduced to two cases: "ret != 0, cannot make smooth progress, unconditional retries may livelock" and "ret ==0, makes some progress, safe to retry".

Memory offline is designed to be able to stop by Ctrl-C. And it has timeout of 120 sec.

I don't called as livelock.

Thanks,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#11 lore raw
Quoted 25 lines · KAMEZAWA Hiroyuki
Quoted 22 lines · On Mon, 25 Oct 2010 12:06:04 +0800 Wu Fengguang <[email protected]> wrote:
Quoted 21 lines · KAMEZAWA Hiroyuki
Quoted 18 lines · On Mon, 25 Oct 2010 11:48:33 +0800 Wu Fengguang <[email protected]> wrote:
Quoted 17 lines · KAMEZAWA Hiroyuki
Quoted 11 lines · On Mon, 25 Oct 2010 12:05:50 +0900 KAMEZAWA Hiroyuki…

This changes behavior.

This "ret" can be > 0 because migrate_page()'s return code is "Return: Number of pages not migrated or error code."

Then,
ret < 0 ===> maybe ebusy
ret > 0 ===> some pages are not migrated. maybe PG_writeback or some ret == 0 ===> ok, all condition green. try next chunk soon.

Then, I added "yield()" and --retrym_max for !ret cases.

                                               ^^^^^^^^
						wrong.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 3 lines

The code here does

ret == 0 ==> ok, all condition green, try next chunk.

It seems reasonable to remove the drain operations for "ret == 0" case. That would help large NUMA boxes noticeably I guess.

Maybe.

OK, I'll post a patch for it.

Quoted 27 lines
Quoted 27 lines
Quoted 9 lines
Quoted 6 lines
Quoted 2 lines

ret > 0 ==> all pages are isolated but some pages cannot be migrated. maybe under I/O
do yield.

Don't know how to deal with the possible "migration fail" pages -- sorry I have no idea about that situation at all.

In typical case, page_count() > 0 by get_user_pages() or PG_writeback is set. All we can do is just waiting.

OK.

Quoted 5 lines
Quoted 2 lines

Perhaps, OOM while offlining pages?

I never see that..because memory offline is scheduled to be done only when there are free memory.

OK.

On OOM migrate_page() will return -ENOMEM, which will be handled in the "ret < 0" case. So it will give up after some retries.

migrate_page() has a comment /* Permanent failure */ when returning positive ret. So it looks safer not to retry indefinitely on the "ret > 0" case?

Quoted 8 lines
Quoted 4 lines

Then it's reduced to two cases: "ret != 0, cannot make smooth progress, unconditional retries may livelock" and "ret ==0, makes some progress, safe to retry".

Memory offline is designed to be able to stop by Ctrl-C. And it has timeout of 120 sec.

I don't called as livelock.

Ah sorry for overlooking that! I should really think twice..(after thinking twice) I find it's even better. Unmigratible pages will be put back to LRU. Then -EBUSY will be returned when trying to isolate it the next time. So it's an imaginary problem.

Thanks,
Fengguang

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

author
#12 lore raw
In reply to Wu Fengguang
Quoted 31 lines · Wu Fengguang
Quoted 13 lines · KAMEZAWA Hiroyuki
Quoted 10 lines · On Mon, 25 Oct 2010 12:05:50 +0900 KAMEZAWA Hiroyuki…

This changes behavior.

This "ret" can be > 0 because migrate_page()'s return code is "Return: Number of pages not migrated or error code."

Then,
ret < 0  ===> maybe ebusy
ret > 0  ===> some pages are not migrated. maybe PG_writeback or some ret == 0 ===> ok, all condition green. try next chunk soon.

Quoted 7 lines
Quoted 1 line

Then, I added "yield()" and --retrym_max for !ret cases.

                                               ^^^^^^^^ wrong.

The code here does

ret == 0 ==> ok, all condition green, try next chunk.

It seems reasonable to remove the drain operations for "ret == 0" case.  That would help large NUMA boxes noticeably I guess.

Quoted 2 lines

ret > 0  ==> all pages are isolated but some pages cannot be migrated. maybe under I/O do yield.

Don't know how to deal with the possible "migration fail" pages -- sorry I have no idea about that situation at all.

Quoted 9 lines

Perhaps, OOM while offlining pages?

Quoted 1 line

ret < 0  ==> some pages may not be able to be isolated. reduce retrycount and yield()

Makes good sense.

Thanks,
Fengguang

Hi, Wu

What about acking these two patches first which doesn't change logic too much. [PATCH 2/3] do_migrate_range: exit loop if not_managed is true. [PATCH 3/3] do_migrate_range: reduce list_empty() check.

For the current, I think more work&discussion is needed and make a clean one is better.
Thanks.

Signature · 8 lines
Regards,
--Bob

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#13 lore raw
In reply to KAMEZAWA Hiroyuki
Quoted 20 lines · KAMEZAWA Hiroyuki
Quoted 17 lines · On Mon, 25 Oct 2010 10:57:03 +0800 Wu Fengguang <[email protected]> wrote:
Quoted 15 lines · KAMEZAWA Hiroyuki
Quoted 10 lines · On Mon, 25 Oct 2010 10:47:31 +0800 Bob Liu <[email protected]> wrote:

It's normal for isolate_lru_page() to fail at times. The failures are typically temporal and may well go away when offline_pages() retries the call. So it seems more reasonable to migrate as much as possible to increase the chance of complete success in next retry.

This patch remove page_count() check and remove putback_lru_pages() and call migrate_pages() regardless of not_managed to reduce failure as much as possible.

-EBUSY should be returned.

Quoted 7 lines
Quoted 5 lines

It does return -EBUSY when ALL pages cannot be isolated from LRU (or is non-LRU pages at all). That means offline_pages() will repeat calls to do_migrate_range() as fast as possible as long as it can make progress.

I read the patch wrong ? "ret = -EBUSY" is dropped and "ret" will be 0 or just a return code of migrate_page().

        for () {
                ret = isolate_lru_page(page);
        }

        if (list_empty(&source))
                goto out;

out:

        return ret;

So do_migrate_range() will return -EBUSY if the last isolate_lru_page() returns -EBUSY.

Quoted 27 lines
Quoted 27 lines

Is that behavior good enough? It does need some comment for this non-obvious return value.

btw, the caller side code can be simplified (no behavior change).

mm/memory_hotplug.c +3 -7
index dd186c1..606d358 100644
@@ -848,17 +848,13 @@ repeat:
    pfn = scan_lru_pages(start_pfn, end_pfn);
    if (pfn) { /* We have page on LRU */
            ret = do_migrate_range(pfn, end_pfn);
-           if (!ret) {
-                   drain = 1;
-                   goto repeat;
-           } else {
-                   if (ret < 0)
-                           if (--retry_max == 0)
-                                   goto failed_removal;
+           if (ret < 0) {
+                   if (--retry_max <= 0)
+                           goto failed_removal;
                    yield();
Quoted 8 lines

mm/memory_hotplug.c repeat: · line 858

Quoted 6 lines
mm/memory_hotplug.c +2 -2
-                   drain = 1;
-                   goto repeat;
            }
+           drain = 1;
+           goto repeat;
    }

This changes behavior.

Ah yes!

Quoted 9 lines

This "ret" can be > 0 because migrate_page()'s return code is "Return: Number of pages not migrated or error code."

Then,
ret < 0 ===> maybe ebusy
ret > 0 ===> some pages are not migrated. maybe PG_writeback or some ret == 0 ===> ok, all condition green. try next chunk soon.

Then, I added "yield()" and --retrym_max for !ret cases.

You are right, there is the "ret > 0, some pages are not migrated" case. But I'm not sure it's PG_writeback pages, because migrate_pages() will wait on writeback after pass 2.

Thanks,
Fengguang

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#14 lore raw
Quoted 36 lines · On Mon, 25 Oct 2010 11:28:27 +0800 Wu Fengguang <[email protected]> wrote:
Quoted 20 lines · KAMEZAWA Hiroyuki
Quoted 17 lines · On Mon, 25 Oct 2010 10:57:03 +0800 Wu Fengguang <[email protected]> wrote:
Quoted 15 lines · KAMEZAWA Hiroyuki
Quoted 10 lines · On Mon, 25 Oct 2010 10:47:31 +0800 Bob Liu <[email protected]> wrote:

It's normal for isolate_lru_page() to fail at times. The failures are typically temporal and may well go away when offline_pages() retries the call. So it seems more reasonable to migrate as much as possible to increase the chance of complete success in next retry.

This patch remove page_count() check and remove putback_lru_pages() and call migrate_pages() regardless of not_managed to reduce failure as much as possible.

-EBUSY should be returned.

Quoted 7 lines
Quoted 5 lines

It does return -EBUSY when ALL pages cannot be isolated from LRU (or is non-LRU pages at all). That means offline_pages() will repeat calls to do_migrate_range() as fast as possible as long as it can make progress.

I read the patch wrong ? "ret = -EBUSY" is dropped and "ret" will be 0 or just a return code of migrate_page().

        for () {
                ret = isolate_lru_page(page);
        }

        if (list_empty(&source))
                goto out;
Quoted 6 lines

out:

        return ret;

So do_migrate_range() will return -EBUSY if the last isolate_lru_page() returns -EBUSY.

Then this patch should be put onto "immediately quit loop when not_managed++" patch. Please write it in description or make a patch series which other guys can see it.

Thanks,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>