Unending loop in __alloc_pages_slowpath following OOM-kill; rfc: patch.

I believe I found a problem in __alloc_pages_slowpath, which allows a process to get stuck endlessly looping, even when lots of memory is available.

Running an I/O and memory intensive stress-test I see a 0-order page allocation with __GFP_IO and __GFP_WAIT, running on a system with very little free memory. Right about the same time that the stress-test gets killed by the OOM-killer, the utility trying to allocate memory gets stuck in __alloc_pages_slowpath even though most of the systems memory was freed by the oom-kill of the stress-test.

The utility ends up looping from the rebalance label down through the wait_iff_congested continiously. Because order=0, __alloc_pages_direct_compact skips the call to get_page_from_freelist. Because all of the reclaimable memory on the system has already been reclaimed, __alloc_pages_direct_reclaim skips the call to get_page_from_freelist. Since there is no __GFP_FS flag, the block with __alloc_pages_may_oom is skipped. The loop hits the wait_iff_congested, then jumps back to rebalance without ever trying to get_page_from_freelist. This loop repeats infinitely.

Is there a reason that this loop is set up this way for 0 order allocations? I applied the below patch, and the problem corrects itself. Does anyone have any thoughts on the patch, or on a better way to address this situation?

The test case is pretty pathological. Running a mix of I/O stress-tests that do a lot of fork() and consume all of the system memory, I can pretty reliably hit this on 600 nodes, in about 12 hours. 32GB/node.

Thanks
Andrew Barry

---
 mm/page_alloc.c |    5 ++++-

 1 files changed, 4 insertions(+), 1 deletions(-)

mm/page_alloc.c +4 -2

index 9f8a97b..c719664 100644
@@ -2158,7 +2158,10 @@ rebalance:
        if (should_alloc_retry(gfp_mask, order, pages_reclaimed)) {
                /* Wait for some write requests to complete then retry */
                wait_iff_congested(preferred_zone, BLK_RW_ASYNC, HZ/50);
-               goto rebalance;
+               if (did_some_progress)
+                       goto rebalance;
+               else
+                       goto restart;
        } else {
                /*
                 * High-order allocations do not necessarily loop after
 
--

To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Quoted 18 lines · Andrew Barry

I believe I found a problem in __alloc_pages_slowpath, which allows a process to get stuck endlessly looping, even when lots of memory is available.

Running an I/O and memory intensive stress-test I see a 0-order page allocation with __GFP_IO and __GFP_WAIT, running on a system with very little free memory. Right about the same time that the stress-test gets killed by the OOM-killer, the utility trying to allocate memory gets stuck in __alloc_pages_slowpath even though most of the systems memory was freed by the oom-kill of the stress-test.

The utility ends up looping from the rebalance label down through the wait_iff_congested continiously. Because order=0, __alloc_pages_direct_compact skips the call to get_page_from_freelist. Because all of the reclaimable memory on the system has already been reclaimed, __alloc_pages_direct_reclaim skips the call to get_page_from_freelist. Since there is no __GFP_FS flag, the block with __alloc_pages_may_oom is skipped. The loop hits the wait_iff_congested, then jumps back to rebalance without ever trying to get_page_from_freelist. This loop repeats infinitely.

Quoted 8 lines

Is there a reason that this loop is set up this way for 0 order allocations? I applied the below patch, and the problem corrects itself. Does anyone have any thoughts on the patch, or on a better way to address this situation?

The test case is pretty pathological. Running a mix of I/O stress-tests that do a lot of fork() and consume all of the system memory, I can pretty reliably hit this on 600 nodes, in about 12 hours. 32GB/node.

It's amazing.
I think it's _very_ rare but it's possible if test program killed by oom has only lots of anonymous pages and allocation tasks try to allocate order-0 page with GFP_NOFS.

When the [in]active lists are empty suddenly(But I am not sure how come the situation happens.)

and we are reclaiming order-0 page,
compaction and __alloc_pages_direct_reclaim doesn't work. compaction doesn't work as it's order-0 page reclaiming. In case of __alloc_pages_direct_reclaim, it would work only if we have lru pages in [in]active list. But unfortunately we don't have any pages in lru list.
So, last resort is following codes in do_try_to_free_pages.

        /* top priority shrink_zones still had more to do? don't OOM, then */
        if (scanning_global_lru(sc) && !all_unreclaimable(zonelist, sc))
                return 1;

But it has a problem, too. all_unreclaimable checks zone->all_unreclaimable. zone->all_unreclaimable is set by below condition.

zone->pages_scanned < zone_reclaimable_pages(zone) * 6

If lru list is completely empty, shrink_zone doesn't work so zone->pages_scanned would be zero. But as we know, zone_page_state isn't exact by per_cpu_pageset. So it might be positive value. After all, zone_reclaimable always return true. It means kswapd never set zone->all_unreclaimable. So last resort become nop.

In this case, current allocation doesn't have a chance to call get_page_from_freelist as Andrew Barry said.

Does it make sense?
If it is, how about this?

Quoted 18 lines · Andrew Barry

I believe I found a problem in __alloc_pages_slowpath, which allows a process to get stuck endlessly looping, even when lots of memory is available.

Running an I/O and memory intensive stress-test I see a 0-order page allocation with __GFP_IO and __GFP_WAIT, running on a system with very little free memory. Right about the same time that the stress-test gets killed by the OOM-killer, the utility trying to allocate memory gets stuck in __alloc_pages_slowpath even though most of the systems memory was freed by the oom-kill of the stress-test.

The utility ends up looping from the rebalance label down through the wait_iff_congested continiously. Because order=0, __alloc_pages_direct_compact skips the call to get_page_from_freelist. Because all of the reclaimable memory on the system has already been reclaimed, __alloc_pages_direct_reclaim skips the call to get_page_from_freelist. Since there is no __GFP_FS flag, the block with __alloc_pages_may_oom is skipped. The loop hits the wait_iff_congested, then jumps back to rebalance without ever trying to get_page_from_freelist. This loop repeats infinitely.

Quoted 8 lines

Is there a reason that this loop is set up this way for 0 order allocations? I applied the below patch, and the problem corrects itself. Does anyone have any thoughts on the patch, or on a better way to address this situation?

The test case is pretty pathological. Running a mix of I/O stress-tests that do a lot of fork() and consume all of the system memory, I can pretty reliably hit this on 600 nodes, in about 12 hours. 32GB/node.

It's amazing.
I think it's _very_ rare but it's possible if test program killed by oom has only lots of anonymous pages and allocation tasks try to allocate order-0 page with GFP_NOFS.

When the [in]active lists are empty suddenly(But I am not sure how come the situation happens.) and we are reclaiming order-0 page, compaction and __alloc_pages_direct_reclaim doesn't work. compaction doesn't work as it's order-0 page reclaiming. In case of __alloc_pages_direct_reclaim, it would work only if we have lru pages in [in]active list. But unfortunately we don't have any pages in lru list.
So, last resort is following codes in do_try_to_free_pages.

        /* top priority shrink_zones still had more to do? don't OOM, then */
        if (scanning_global_lru(sc) && !all_unreclaimable(zonelist, sc))
                return 1;

But it has a problem, too. all_unreclaimable checks zone->all_unreclaimable. zone->all_unreclaimable is set by below condition.

zone->pages_scanned < zone_reclaimable_pages(zone) * 6

If lru list is completely empty, shrink_zone doesn't work so zone->pages_scanned would be zero. But as we know, zone_page_state isn't exact by per_cpu_pageset. So it might be positive value. After all, zone_reclaimable always return true. It means kswapd never set zone->all_unreclaimable. So last resort become nop.

In this case, current allocation doesn't have a chance to call get_page_from_freelist as Andrew Barry said.

Does it make sense?
If it is, how about this?

mm/page_alloc.c +1 -0

index ebc7faa..4f64355 100644
@@ -2105,6 +2105,7 @@ restart:
                first_zones_zonelist(zonelist, high_zoneidx, NULL,
                                        &preferred_zone);
 
+rebalance:
        /* This is the last chance, in general, before the goto nopage. */
        page = get_page_from_freelist(gfp_mask, nodemask, order, zonelist,
                        high_zoneidx, alloc_flags & ~ALLOC_NO_WATERMARKS,

Quoted 28 lines · Minchan Kim

Quoted 26 lines · Andrew Barry

I believe I found a problem in __alloc_pages_slowpath, which allows a process to get stuck endlessly looping, even when lots of memory is available.

Running an I/O and memory intensive stress-test I see a 0-order page allocation with __GFP_IO and __GFP_WAIT, running on a system with very little free memory. Right about the same time that the stress-test gets killed by the OOM-killer, the utility trying to allocate memory gets stuck in __alloc_pages_slowpath even though most of the systems memory was freed by the oom-kill of the stress-test.

The utility ends up looping from the rebalance label down through the wait_iff_congested continiously. Because order=0, __alloc_pages_direct_compact skips the call to get_page_from_freelist. Because all of the reclaimable memory on the system has already been reclaimed, __alloc_pages_direct_reclaim skips the call to get_page_from_freelist. Since there is no __GFP_FS flag, the block with __alloc_pages_may_oom is skipped. The loop hits the wait_iff_congested, then jumps back to rebalance without ever trying to get_page_from_freelist. This loop repeats infinitely.

Is there a reason that this loop is set up this way for 0 order allocations? I applied the below patch, and the problem corrects itself. Does anyone have any thoughts on the patch, or on a better way to address this situation?

The test case is pretty pathological. Running a mix of I/O stress-tests that do a lot of fork() and consume all of the system memory, I can pretty reliably hit this on 600 nodes, in about 12 hours. 32GB/node.

Quoted 4 lines

It's amazing.
I think it's _very_ rare but it's possible if test program killed by oom has only lots of anonymous pages and allocation tasks try to allocate order-0 page with GFP_NOFS.

Unfortunately very rare is a subjective thing. We have been hitting it a couple times a week in our test lab.

Quoted 47 lines

When the [in]active lists are empty suddenly(But I am not sure how come the situation happens.) and we are reclaiming order-0 page, compaction and __alloc_pages_direct_reclaim doesn't work. compaction doesn't work as it's order-0 page reclaiming.  In case of __alloc_pages_direct_reclaim, it would work only if we have lru pages in [in]active list. But unfortunately we don't have any pages in lru list.
So, last resort is following codes in do_try_to_free_pages.

        /* top priority shrink_zones still had more to do? don't OOM, then */ if (scanning_global_lru(sc) && !all_unreclaimable(zonelist, sc)) return 1;

But it has a problem, too. all_unreclaimable checks zone->all_unreclaimable. zone->all_unreclaimable is set by below condition.

zone->pages_scanned < zone_reclaimable_pages(zone) * 6

If lru list is completely empty, shrink_zone doesn't work so zone->pages_scanned would be zero. But as we know, zone_page_state isn't exact by per_cpu_pageset. So it might be positive value. After all, zone_reclaimable always return true. It means kswapd never set zone->all_unreclaimable.  So last resort become nop.

In this case, current allocation doesn't have a chance to call get_page_from_freelist as Andrew Barry said.

Does it make sense?
If it is, how about this?

mm/page_alloc.c +1 -0

index ebc7faa..4f64355 100644
@@ -2105,6 +2105,7 @@ restart:
                 first_zones_zonelist(zonelist, high_zoneidx, NULL,
                                         &preferred_zone);
 
+rebalance:
         /* This is the last chance, in general, before the goto nopage. */
         page = get_page_from_freelist(gfp_mask, nodemask, order, zonelist,
                         high_zoneidx, alloc_flags & ~ALLOC_NO_WATERMARKS,

@@ -2112,7 +2113,6 @@ restart:
         if (page)
                 goto got_pg;
 

mm/page_alloc.c restart: · line 2118

Quoted 4 lines

mm/page_alloc.c +0 -1

-rebalance:
         /* Allocate without watermarks if the context allows */
         if (alloc_flags & ALLOC_NO_WATERMARKS) {
                 page = __alloc_pages_high_priority(gfp_mask, order,

I think your solution is simpler than my patch.
Thanks very much.

-Andrew

Hi Andrew,

Quoted 29 lines · Andrew Barry

Quoted 28 lines · Minchan Kim

Quoted 26 lines · Andrew Barry

I believe I found a problem in __alloc_pages_slowpath, which allows a process to get stuck endlessly looping, even when lots of memory is available.

Running an I/O and memory intensive stress-test I see a 0-order page allocation with __GFP_IO and __GFP_WAIT, running on a system with very little free memory. Right about the same time that the stress-test gets killed by the OOM-killer, the utility trying to allocate memory gets stuck in __alloc_pages_slowpath even though most of the systems memory was freed by the oom-kill of the stress-test.

The utility ends up looping from the rebalance label down through the wait_iff_congested continiously. Because order=0, __alloc_pages_direct_compact skips the call to get_page_from_freelist. Because all of the reclaimable memory on the system has already been reclaimed, __alloc_pages_direct_reclaim skips the call to get_page_from_freelist. Since there is no __GFP_FS flag, the block with __alloc_pages_may_oom is skipped. The loop hits the wait_iff_congested, then jumps back to rebalance without ever trying to get_page_from_freelist. This loop repeats infinitely.

Is there a reason that this loop is set up this way for 0 order allocations? I applied the below patch, and the problem corrects itself. Does anyone have any thoughts on the patch, or on a better way to address this situation?

The test case is pretty pathological. Running a mix of I/O stress-tests that do a lot of fork() and consume all of the system memory, I can pretty reliably hit this on 600 nodes, in about 12 hours. 32GB/node.

Quoted 7 lines

Quoted 4 lines

It's amazing.
I think it's _very_ rare but it's possible if test program killed by oom has only lots of anonymous pages and allocation tasks try to allocate order-0 page with GFP_NOFS.

Unfortunately very rare is a subjective thing. We have been hitting it a couple times a week in our test lab.

Okay.

Quoted 47 lines

Quoted 47 lines

When the [in]active lists are empty suddenly(But I am not sure how come the situation happens.) and we are reclaiming order-0 page, compaction and __alloc_pages_direct_reclaim doesn't work. compaction doesn't work as it's order-0 page reclaiming.  In case of __alloc_pages_direct_reclaim, it would work only if we have lru pages in [in]active list. But unfortunately we don't have any pages in lru list.
So, last resort is following codes in do_try_to_free_pages.

        /* top priority shrink_zones still had more to do? don't OOM, then */ if (scanning_global_lru(sc) && !all_unreclaimable(zonelist, sc)) return 1;

But it has a problem, too. all_unreclaimable checks zone->all_unreclaimable. zone->all_unreclaimable is set by below condition.

zone->pages_scanned < zone_reclaimable_pages(zone) * 6

If lru list is completely empty, shrink_zone doesn't work so zone->pages_scanned would be zero. But as we know, zone_page_state isn't exact by per_cpu_pageset. So it might be positive value. After all, zone_reclaimable always return true. It means kswapd never set zone->all_unreclaimable.  So last resort become nop.

In this case, current allocation doesn't have a chance to call get_page_from_freelist as Andrew Barry said.

Does it make sense?
If it is, how about this?

mm/page_alloc.c +1 -0

index ebc7faa..4f64355 100644
@@ -2105,6 +2105,7 @@ restart:
                 first_zones_zonelist(zonelist, high_zoneidx, NULL,
                                         &preferred_zone);
 
+rebalance:
         /* This is the last chance, in general, before the goto nopage. */
         page = get_page_from_freelist(gfp_mask, nodemask, order, zonelist,
                         high_zoneidx, alloc_flags & ~ALLOC_NO_WATERMARKS,

@@ -2112,7 +2113,6 @@ restart:
         if (page)
                 goto got_pg;
 

Quoted 7 lines

mm/page_alloc.c restart: · line 2118

Quoted 4 lines

mm/page_alloc.c +0 -1

-rebalance:
         /* Allocate without watermarks if the context allows */
         if (alloc_flags & ALLOC_NO_WATERMARKS) {
                 page = __alloc_pages_high_priority(gfp_mask, order,

I think your solution is simpler than my patch.
Thanks very much.

You find the problem and it's harder than fix, I think. So I think you have to get a credit.

Could you send the patch to akpm with Cced Mel and me? (Maybe it's the subject to send stable).
You can get my Reviewed-by.

Thanks for the good bug reporting.

From 8bd3f16736548375238161d1bd85f7d7c381031f Mon Sep 17 00:00:00 2001

Quoted 8 lines

-Andrew

Signature · 2 lines

Kind regards,
Minchan Kim

From: Andrew Barry<[email protected]>

I believe I found a problem in __alloc_pages_slowpath, which allows a process to get stuck endlessly looping, even when lots of memory is available.

Signed-off-by Andrew Barry<[email protected]>

Reviewed-by Minchan Kim<[email protected]>

Cc Mel Gorman<[email protected]>

<SNIP>

From 8bd3f16736548375238161d1bd85f7d7c381031f Mon Sep 17 00:00:00 2001
From: Minchan Kim <[email protected]>
Date: Sat, 21 May 2011 01:37:41 +0900
Subject: [PATCH] Prevent unending loop in __alloc_pages_slowpath

From: Andrew Barry <[email protected]>

I believe I found a problem in __alloc_pages_slowpath, which allows a process to get stuck endlessly looping, even when lots of memory is available.

Running an I/O and memory intensive stress-test I see a 0-order page allocation with __GFP_IO and __GFP_WAIT, running on a system with very little free memory. Right about the same time that the stress-test gets killed by the OOM-killer, the utility trying to allocate memory gets stuck in __alloc_pages_slowpath even though most of the systems memory was freed by the oom-kill of the stress-test.

The utility ends up looping from the rebalance label down through the wait_iff_congested continiously. Because order=0, __alloc_pages_direct_compact skips the call to get_page_from_freelist. Because all of the reclaimable memory on the system has already been reclaimed, __alloc_pages_direct_reclaim skips the call to get_page_from_freelist. Since there is no __GFP_FS flag, the block with __alloc_pages_may_oom is skipped. The loop hits the wait_iff_congested, then jumps back to rebalance without ever trying to get_page_from_freelist. This loop repeats infinitely.

The test case is pretty pathological. Running a mix of I/O stress-tests that do a lot of fork() and consume all of the system memory, I can pretty reliably hit this on 600 nodes, in about 12 hours. 32GB/node.

Signed-off-by Andrew Barry <[email protected]>

Reviewed-by Minchan Kim <[email protected]>

Cc Mel Gorman <[email protected]>

Quoted 1 line

From 8bd3f16736548375238161d1bd85f7d7c381031f Mon Sep 17 00:00:00 2001

From: Minchan Kim <[email protected]>
Date: Sat, 21 May 2011 01:37:41 +0900
Subject: [PATCH] Prevent unending loop in __alloc_pages_slowpath

From: Andrew Barry <[email protected]>

I believe I found a problem in __alloc_pages_slowpath, which allows a process to get stuck endlessly looping, even when lots of memory is available.

Running an I/O and memory intensive stress-test I see a 0-order page allocation with __GFP_IO and __GFP_WAIT, running on a system with very little free memory. Right about the same time that the stress-test gets killed by the OOM-killer, the utility trying to allocate memory gets stuck in __alloc_pages_slowpath even though most of the systems memory was freed by the oom-kill of the stress-test.

The utility ends up looping from the rebalance label down through the wait_iff_congested continiously. Because order=0, __alloc_pages_direct_compact skips the call to get_page_from_freelist. Because all of the reclaimable memory on the system has already been reclaimed, __alloc_pages_direct_reclaim skips the call to get_page_from_freelist. Since there is no __GFP_FS flag, the block with __alloc_pages_may_oom is skipped. The loop hits the wait_iff_congested, then jumps back to rebalance without ever trying to get_page_from_freelist. This loop repeats infinitely.

The test case is pretty pathological. Running a mix of I/O stress-tests that do a lot of fork() and consume all of the system memory, I can pretty reliably hit this on 600 nodes, in about 12 hours. 32GB/node.

Signed-off-by Andrew Barry <[email protected]>

Reviewed-by Minchan Kim <[email protected]>

Cc Mel Gorman <[email protected]>

---
 mm/page_alloc.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

mm/page_alloc.c +1 -0

index 3f8bce2..e78b324 100644
@@ -2064,6 +2064,7 @@ restart:
 		first_zones_zonelist(zonelist, high_zoneidx, NULL,
 					&preferred_zone);
 
+rebalance:
 	/* This is the last chance, in general, before the goto nopage. */
 	page = get_page_from_freelist(gfp_mask, nodemask, order, zonelist,
 			high_zoneidx, alloc_flags & ~ALLOC_NO_WATERMARKS,

From 2e77784668f6ca53d88ecb46aa6b99d9d0f33ffa Mon Sep 17 00:00:00 2001

Quoted 49 lines

Quoted 1 line

From 8bd3f16736548375238161d1bd85f7d7c381031f Mon Sep 17 00:00:00 2001

From: Minchan Kim <[email protected]>
Date: Sat, 21 May 2011 01:37:41 +0900
Subject: [PATCH] Prevent unending loop in __alloc_pages_slowpath

From: Andrew Barry <[email protected]>

I believe I found a problem in __alloc_pages_slowpath, which allows a process to get stuck endlessly looping, even when lots of memory is available.

Running an I/O and memory intensive stress-test I see a 0-order page allocation with __GFP_IO and __GFP_WAIT, running on a system with very little free memory. Right about the same time that the stress-test gets killed by the OOM-killer, the utility trying to allocate memory gets stuck in __alloc_pages_slowpath even though most of the systems memory was freed by the oom-kill of the stress-test.

The utility ends up looping from the rebalance label down through the wait_iff_congested continiously. Because order=0, __alloc_pages_direct_compact skips the call to get_page_from_freelist. Because all of the reclaimable memory on the system has already been reclaimed, __alloc_pages_direct_reclaim skips the call to get_page_from_freelist. Since there is no __GFP_FS flag, the block with __alloc_pages_may_oom is skipped. The loop hits the wait_iff_congested, then jumps back to rebalance without ever trying to get_page_from_freelist. This loop repeats infinitely.

The test case is pretty pathological. Running a mix of I/O stress-tests that do a lot of fork() and consume all of the system memory, I can pretty reliably hit this on 600 nodes, in about 12 hours. 32GB/node.

Signed-off-by Andrew Barry <[email protected]>

Reviewed-by Minchan Kim <[email protected]>

Cc Mel Gorman <[email protected]>

---
 mm/page_alloc.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

mm/page_alloc.c +1 -0

index 3f8bce2..e78b324 100644
@@ -2064,6 +2064,7 @@ restart:
 		first_zones_zonelist(zonelist, high_zoneidx, NULL,
 					&preferred_zone);
 
+rebalance:
 	/* This is the last chance, in general, before the goto nopage. */
 	page = get_page_from_freelist(gfp_mask, nodemask, order, zonelist,
 			high_zoneidx, alloc_flags & ~ALLOC_NO_WATERMARKS,

mm/page_alloc.c restart: · line 2077

Quoted 7 lines

mm/page_alloc.c +0 -1

@@ -2071,7 +2072,6 @@ restart:
 	if (page)
 		goto got_pg;
 
-rebalance:
 	/* Allocate without watermarks if the context allows */
 	if (alloc_flags & ALLOC_NO_WATERMARKS) {
 		page = __alloc_pages_high_priority(gfp_mask, order,

I'm sorry I missed this thread long time.

In this case, I think we should call drain_all_pages(). then following patch is better.
However I also think your patch is valuable. because while the task is sleeping in wait_iff_congested(), an another task may free some pages. thus, rebalance path should try to get free pages. iow, you makes sense.

So, I'd like to propose to merge both your and my patch.

Quoted 52 lines

Quoted 1 line

From 8bd3f16736548375238161d1bd85f7d7c381031f Mon Sep 17 00:00:00 2001

From: Minchan Kim <[email protected]>
Date: Sat, 21 May 2011 01:37:41 +0900
Subject: [PATCH] Prevent unending loop in __alloc_pages_slowpath

From: Andrew Barry <[email protected]>

I believe I found a problem in __alloc_pages_slowpath, which allows a process to get stuck endlessly looping, even when lots of memory is available.

Running an I/O and memory intensive stress-test I see a 0-order page allocation with __GFP_IO and __GFP_WAIT, running on a system with very little free memory. Right about the same time that the stress-test gets killed by the OOM-killer, the utility trying to allocate memory gets stuck in __alloc_pages_slowpath even though most of the systems memory was freed by the oom-kill of the stress-test.

The utility ends up looping from the rebalance label down through the wait_iff_congested continiously. Because order=0, __alloc_pages_direct_compact skips the call to get_page_from_freelist. Because all of the reclaimable memory on the system has already been reclaimed, __alloc_pages_direct_reclaim skips the call to get_page_from_freelist. Since there is no __GFP_FS flag, the block with __alloc_pages_may_oom is skipped. The loop hits the wait_iff_congested, then jumps back to rebalance without ever trying to get_page_from_freelist. This loop repeats infinitely.

The test case is pretty pathological. Running a mix of I/O stress-tests that do a lot of fork() and consume all of the system memory, I can pretty reliably hit this on 600 nodes, in about 12 hours. 32GB/node.

Signed-off-by Andrew Barry <[email protected]>

Reviewed-by Minchan Kim <[email protected]>

Cc Mel Gorman <[email protected]>

---
 mm/page_alloc.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

mm/page_alloc.c +1 -0

index 3f8bce2..e78b324 100644
@@ -2064,6 +2064,7 @@ restart:
 		first_zones_zonelist(zonelist, high_zoneidx, NULL,
 					&preferred_zone);
 
+rebalance:
 	/* This is the last chance, in general, before the goto nopage. */
 	page = get_page_from_freelist(gfp_mask, nodemask, order, zonelist,
 			high_zoneidx, alloc_flags & ~ALLOC_NO_WATERMARKS,

@@ -2071,7 +2072,6 @@ restart:
 	if (page)
 		goto got_pg;
 

mm/page_alloc.c restart: · line 2077

Quoted 4 lines

mm/page_alloc.c +0 -1

-rebalance:
 	/* Allocate without watermarks if the context allows */
 	if (alloc_flags & ALLOC_NO_WATERMARKS) {
 		page = __alloc_pages_high_priority(gfp_mask, order,

I'm sorry I missed this thread long time.

In this case, I think we should call drain_all_pages().

then following
patch is better.
However I also think your patch is valuable. because while the task is sleeping in wait_iff_congested(), an another task may free some pages. thus, rebalance path should try to get free pages. iow, you makes sense.

So, I'd like to propose to merge both your and my patch.

Thanks.

From 2e77784668f6ca53d88ecb46aa6b99d9d0f33ffa Mon Sep 17 00:00:00 2001
From: KOSAKI Motohiro <[email protected]>
Date: Tue, 24 May 2011 13:41:57 +0900
Subject: [PATCH] vmscan: remove painful micro optimization

Currently, __alloc_pages_direct_reclaim() call get_page_from_freelist() only if try_to_free_pages() return !0.

It's no necessary micro optimization becauase "return 0" mean vmscan reached priority 0 and didn't get any pages, iow, it's really slow path. But also it has bad side effect. If we don't call drain_all_pages(), we have a chance to get infinite loop.

This patch remove its bad and meaningless micro optimization.

Signed-off-by KOSAKI Motohiro <[email protected]>

---
 mm/page_alloc.c |    3 ---
 1 files changed, 0 insertions(+), 3 deletions(-)

mm/page_alloc.c +0 -3

index 1572079..c41d488 100644
@@ -1950,9 +1950,6 @@ __alloc_pages_direct_reclaim(gfp_t gfp_mask, unsigned int order,
 
 	cond_resched();
 
-	if (unlikely(!(*did_some_progress)))
-		return NULL;
-
 retry:
 	page = get_page_from_freelist(gfp_mask, nodemask, order,
 					zonelist, high_zoneidx,

Signature · 1 line

1.7.3.1

Quoted 59 lines · KOSAKI Motohiro

Quoted 49 lines

Quoted 1 line

From 8bd3f16736548375238161d1bd85f7d7c381031f Mon Sep 17 00:00:00 2001

From: Minchan Kim <[email protected]>
Date: Sat, 21 May 2011 01:37:41 +0900
Subject: [PATCH] Prevent unending loop in __alloc_pages_slowpath

From: Andrew Barry <[email protected]>

I believe I found a problem in __alloc_pages_slowpath, which allows a process to get stuck endlessly looping, even when lots of memory is available.

Running an I/O and memory intensive stress-test I see a 0-order page allocation with __GFP_IO and __GFP_WAIT, running on a system with very little free memory. Right about the same time that the stress-test gets killed by the OOM-killer, the utility trying to allocate memory gets stuck in __alloc_pages_slowpath even though most of the systems memory was freed by the oom-kill of the stress-test.

The utility ends up looping from the rebalance label down through the wait_iff_congested continiously. Because order=0, __alloc_pages_direct_compact skips the call to get_page_from_freelist. Because all of the reclaimable memory on the system has already been reclaimed, __alloc_pages_direct_reclaim skips the call to get_page_from_freelist. Since there is no __GFP_FS flag, the block with __alloc_pages_may_oom is skipped. The loop hits the wait_iff_congested, then jumps back to rebalance without ever trying to get_page_from_freelist. This loop repeats infinitely.

The test case is pretty pathological. Running a mix of I/O stress-tests that do a lot of fork() and consume all of the system memory, I can pretty reliably hit this on 600 nodes, in about 12 hours. 32GB/node.

Signed-off-by Andrew Barry <[email protected]>

Reviewed-by Minchan Kim <[email protected]>

Cc Mel Gorman <[email protected]>

---
 mm/page_alloc.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

mm/page_alloc.c +1 -0

index 3f8bce2..e78b324 100644
@@ -2064,6 +2064,7 @@ restart:
 		first_zones_zonelist(zonelist, high_zoneidx, NULL,
 					&preferred_zone);
 
+rebalance:
 	/* This is the last chance, in general, before the goto nopage. */
 	page = get_page_from_freelist(gfp_mask, nodemask, order, zonelist,
 			high_zoneidx, alloc_flags & ~ALLOC_NO_WATERMARKS,

mm/page_alloc.c restart: · line 2077

Quoted 7 lines

mm/page_alloc.c +0 -1

@@ -2071,7 +2072,6 @@ restart:
 	if (page)
 		goto got_pg;
 
-rebalance:
 	/* Allocate without watermarks if the context allows */
 	if (alloc_flags & ALLOC_NO_WATERMARKS) {
 		page = __alloc_pages_high_priority(gfp_mask, order,

I'm sorry I missed this thread long time.

Quoted 1 line

In this case, I think we should call drain_all_pages().

Why?

If the direct reclaimer failed to reclaim any pages on its own, the call to get_page_from_freelist() is going to be useless and there is no guarantee that any other CPU managed to reclaim pages either. All this ends up doing is sending in IPI which if it's very lucky will take a page from another CPUs free list.

Quoted 20 lines

then following
patch is better.
However I also think your patch is valuable. because while the task is sleeping in wait_iff_congested(), an another task may free some pages. thus, rebalance path should try to get free pages. iow, you makes sense.

So, I'd like to propose to merge both your and my patch.

Thanks.

From 2e77784668f6ca53d88ecb46aa6b99d9d0f33ffa Mon Sep 17 00:00:00 2001
From: KOSAKI Motohiro <[email protected]>
Date: Tue, 24 May 2011 13:41:57 +0900
Subject: [PATCH] vmscan: remove painful micro optimization

Currently, __alloc_pages_direct_reclaim() call get_page_from_freelist() only if try_to_free_pages() return !0.

Quoted 5 lines

It's no necessary micro optimization becauase "return 0" mean vmscan reached priority 0 and didn't get any pages, iow, it's really slow path. But also it has bad side effect. If we don't call drain_all_pages(), we have a chance to get infinite loop.

With the "rebalance" patch, where is the infinite loop?

Quoted 60 lines · Mel Gorman

Quoted 59 lines · KOSAKI Motohiro

Quoted 49 lines

Quoted 1 line

From 8bd3f16736548375238161d1bd85f7d7c381031f Mon Sep 17 00:00:00 2001

From: Minchan Kim <[email protected]>
Date: Sat, 21 May 2011 01:37:41 +0900
Subject: [PATCH] Prevent unending loop in __alloc_pages_slowpath

From: Andrew Barry <[email protected]>

I believe I found a problem in __alloc_pages_slowpath, which allows a process to get stuck endlessly looping, even when lots of memory is available.

Running an I/O and memory intensive stress-test I see a 0-order page allocation with __GFP_IO and __GFP_WAIT, running on a system with very little free memory. Right about the same time that the stress-test gets killed by the OOM-killer, the utility trying to allocate memory gets stuck in __alloc_pages_slowpath even though most of the systems memory was freed by the oom-kill of the stress-test.

The utility ends up looping from the rebalance label down through the wait_iff_congested continiously. Because order=0, __alloc_pages_direct_compact skips the call to get_page_from_freelist. Because all of the reclaimable memory on the system has already been reclaimed, __alloc_pages_direct_reclaim skips the call to get_page_from_freelist. Since there is no __GFP_FS flag, the block with __alloc_pages_may_oom is skipped. The loop hits the wait_iff_congested, then jumps back to rebalance without ever trying to get_page_from_freelist. This loop repeats infinitely.

The test case is pretty pathological. Running a mix of I/O stress-tests that do a lot of fork() and consume all of the system memory, I can pretty reliably hit this on 600 nodes, in about 12 hours. 32GB/node.

Signed-off-by Andrew Barry <[email protected]>

Reviewed-by Minchan Kim <[email protected]>

Cc Mel Gorman <[email protected]>

---
 mm/page_alloc.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

mm/page_alloc.c +1 -0

index 3f8bce2..e78b324 100644
@@ -2064,6 +2064,7 @@ restart:
 		first_zones_zonelist(zonelist, high_zoneidx, NULL,
 					&preferred_zone);
 
+rebalance:
 	/* This is the last chance, in general, before the goto nopage. */
 	page = get_page_from_freelist(gfp_mask, nodemask, order, zonelist,
 			high_zoneidx, alloc_flags & ~ALLOC_NO_WATERMARKS,

mm/page_alloc.c restart: · line 2077

Quoted 7 lines

mm/page_alloc.c +0 -1

@@ -2071,7 +2072,6 @@ restart:
 	if (page)
 		goto got_pg;
 
-rebalance:
 	/* Allocate without watermarks if the context allows */
 	if (alloc_flags & ALLOC_NO_WATERMARKS) {
 		page = __alloc_pages_high_priority(gfp_mask, order,

I'm sorry I missed this thread long time.

Quoted 3 lines

Quoted 1 line

In this case, I think we should call drain_all_pages().

Why?

Otherwise, we don't have good PCP dropping trigger. Big machine might have big pcp cache.

Quoted 6 lines

If the direct reclaimer failed to reclaim any pages on its own, the call to get_page_from_freelist() is going to be useless and there is no guarantee that any other CPU managed to reclaim pages either. All this ends up doing is sending in IPI which if it's very lucky will take a page from another CPUs free list.

It's no matter. because did_some_progress==0 mean vmscan failed to reclaim any pages and reach priority==0. Thus, it obviously slow path.

Quoted 25 lines

Quoted 24 lines

then following
patch is better.
However I also think your patch is valuable. because while the task is sleeping in wait_iff_congested(), an another task may free some pages. thus, rebalance path should try to get free pages. iow, you makes sense.

So, I'd like to propose to merge both your and my patch.

Thanks.

From 2e77784668f6ca53d88ecb46aa6b99d9d0f33ffa Mon Sep 17 00:00:00 2001
From: KOSAKI Motohiro <[email protected]>
Date: Tue, 24 May 2011 13:41:57 +0900
Subject: [PATCH] vmscan: remove painful micro optimization

Currently, __alloc_pages_direct_reclaim() call get_page_from_freelist() only if try_to_free_pages() return !0.

It's no necessary micro optimization becauase "return 0" mean vmscan reached priority 0 and didn't get any pages, iow, it's really slow path. But also it has bad side effect. If we don't call drain_all_pages(), we have a chance to get infinite loop.

Quoted 1 line

With the "rebalance" patch, where is the infinite loop?

I wrote the above.

Quoted 5 lines

Quoted 1 line

Why?

Otherwise, we don't have good PCP dropping trigger. Big machine might have big pcp cache.

Big machines also have a large cost for sending IPIs.

Quoted 6 lines

Quoted 5 lines

Quoted 1 line

Why?

Otherwise, we don't have good PCP dropping trigger. Big machine might have big pcp cache.

Quoted 1 line

Big machines also have a large cost for sending IPIs.

Yes. But it's only matter if IPIs are frequently happen.
But, drain_all_pages() is NOT only IPI source. some vmscan function (e.g. try_to_umap) makes a lot of IPIs.

Then, it's _relatively_ not costly. I have a question. Do you compare which operation and drain_all_pages()? IOW, your "costly" mean which scenario suspect?

Quoted 12 lines · KOSAKI Motohiro

Quoted 7 lines

Quoted 5 lines

Quoted 1 line

Why?

Otherwise, we don't have good PCP dropping trigger. Big machine might have big pcp cache.

Big machines also have a large cost for sending IPIs.

Yes. But it's only matter if IPIs are frequently happen.
But, drain_all_pages() is NOT only IPI source. some vmscan function (e.g. try_to_umap) makes a lot of IPIs.

Quoted 3 lines

Then, it's _relatively_ not costly. I have a question. Do you compare which operation and drain_all_pages()? IOW, your "costly" mean which scenario suspect?

I am concerned that if the machine gets into trouble and we are failing to reclaim that sending more IPIs is not going to help any. There is no evidence at the moment that sending extra IPIs here will help anything.

Quoted 13 lines · Mel Gorman

Quoted 12 lines · KOSAKI Motohiro

Quoted 7 lines

Quoted 5 lines

Quoted 1 line

Why?

Otherwise, we don't have good PCP dropping trigger. Big machine might have big pcp cache.

Big machines also have a large cost for sending IPIs.

Yes. But it's only matter if IPIs are frequently happen.
But, drain_all_pages() is NOT only IPI source. some vmscan function (e.g. try_to_umap) makes a lot of IPIs.

Quoted 7 lines

Quoted 3 lines

Then, it's _relatively_ not costly. I have a question. Do you compare which operation and drain_all_pages()? IOW, your "costly" mean which scenario suspect?

I am concerned that if the machine gets into trouble and we are failing to reclaim that sending more IPIs is not going to help any. There is no evidence at the moment that sending extra IPIs here will help anything.

In old days, we always call drain_all_pages() if did_some_progress!=0. But current kernel only call it when get_page_from_freelist() fail. So, wait_iff_congested() may help but no guarantee to help us.

If you still strongly worry about IPI cost, I'm concern to move drain_all_pages() to more unfrequently point. but to ignore pcp makes less sense, IMHO.

Quoted 7 lines

In old days, we always call drain_all_pages() if did_some_progress!=0. But current kernel only call it when get_page_from_freelist() fail. So, wait_iff_congested() may help but no guarantee to help us.

If you still strongly worry about IPI cost, I'm concern to move drain_all_pages() to more unfrequently point. but to ignore pcp makes less sense, IMHO.

Yes, I'm worried about it because excessive time
spent in drain_all_pages() has come up on the past
http://lkml.org/lkml/2010/8/23/81 . The PCP lists are not being ignored at the moment. They are drained when direct reclaim makes forward progress but still fails to allocate a page.

Quoted 52 lines

Quoted 1 line

From 8bd3f16736548375238161d1bd85f7d7c381031f Mon Sep 17 00:00:00 2001

From: Minchan Kim <[email protected]>
Date: Sat, 21 May 2011 01:37:41 +0900
Subject: [PATCH] Prevent unending loop in __alloc_pages_slowpath

From: Andrew Barry <[email protected]>

I believe I found a problem in __alloc_pages_slowpath, which allows a process to get stuck endlessly looping, even when lots of memory is available.

Running an I/O and memory intensive stress-test I see a 0-order page allocation with __GFP_IO and __GFP_WAIT, running on a system with very little free memory. Right about the same time that the stress-test gets killed by the OOM-killer, the utility trying to allocate memory gets stuck in __alloc_pages_slowpath even though most of the systems memory was freed by the oom-kill of the stress-test.

The utility ends up looping from the rebalance label down through the wait_iff_congested continiously. Because order=0, __alloc_pages_direct_compact skips the call to get_page_from_freelist. Because all of the reclaimable memory on the system has already been reclaimed, __alloc_pages_direct_reclaim skips the call to get_page_from_freelist. Since there is no __GFP_FS flag, the block with __alloc_pages_may_oom is skipped. The loop hits the wait_iff_congested, then jumps back to rebalance without ever trying to get_page_from_freelist. This loop repeats infinitely.

The test case is pretty pathological. Running a mix of I/O stress-tests that do a lot of fork() and consume all of the system memory, I can pretty reliably hit this on 600 nodes, in about 12 hours. 32GB/node.

Signed-off-by Andrew Barry <[email protected]>

Reviewed-by Minchan Kim <[email protected]>

Cc Mel Gorman <[email protected]>

---
 mm/page_alloc.c |    2 +-

 1 files changed, 1 insertions(+), 1 deletions(-)

mm/page_alloc.c +1 -0

index 3f8bce2..e78b324 100644
@@ -2064,6 +2064,7 @@ restart:
               first_zones_zonelist(zonelist, high_zoneidx, NULL,
                                       &preferred_zone);
 
+rebalance:
       /* This is the last chance, in general, before the goto nopage. */
       page = get_page_from_freelist(gfp_mask, nodemask, order, zonelist,
                       high_zoneidx, alloc_flags & ~ALLOC_NO_WATERMARKS,

@@ -2071,7 +2072,6 @@ restart:
       if (page)
               goto got_pg;
 

mm/page_alloc.c restart: · line 2077

Quoted 4 lines

mm/page_alloc.c +0 -1

-rebalance:
       /* Allocate without watermarks if the context allows */
       if (alloc_flags & ALLOC_NO_WATERMARKS) {
               page = __alloc_pages_high_priority(gfp_mask, order,

I'm sorry I missed this thread long time.

In this case, I think we should call drain_all_pages(). then following patch is better.

However I also think your patch is valuable. because while the task is sleeping in wait_iff_congested(), an another task may free some pages. thus, rebalance path should try to get free pages. iow, you makes sense.

So, I'd like to propose to merge both your and my patch.

Quoted 1 line

I'm sorry I missed this thread long time.

No problem. It would be better than not review.

Quoted 2 lines

In this case, I think we should call drain_all_pages(). then following patch is better.

Strictly speaking, this problem isn't related to drain_all_pages. This problem caused by lru empty but I admit it could work well if your patch applied.
So yours could help, too.

Quoted 3 lines

However I also think your patch is valuable. because while the task is sleeping in wait_iff_congested(), an another task may free some pages. thus, rebalance path should try to get free pages. iow, you makes sense.

Yes.
Off-topic.
I would like to move cond_resched below get_page_from_freelist in __alloc_pages_direct_reclaim. Otherwise, it is likely we can be stolen pages to other processes.
One more benefit is that if it's apparently OOM path(ie, did_some_progress = 0), we can reduce OOM kill latency due to remove unnecessary cond_resched.

Quoted 1 line

So, I'd like to propose to merge both your and my patch.

Recently, there was discussion on drain_all_pages with Wu. He saw much overhead in 8-core system, AFAIR.
I Cced Wu.

How about checking per-cpu before calling drain_all_pages() than unconditional calling?
if (per_cpu_ptr(zone->pageset, smp_processor_id())

    drain_all_pages();

Of course, It can miss other CPU free pages. But above routine assume local cpu direct reclaim is successful but it failed by per-cpu. So I think it works.

Thanks for good suggestion and Reviewed-by, KOSAKI.

Quoted 3 lines

Quoted 1 line

I'm sorry I missed this thread long time.

No problem. It would be better than not review.

thx.

Quoted 12 lines

Quoted 2 lines

In this case, I think we should call drain_all_pages(). then following patch is better.

Strictly speaking, this problem isn't related to drain_all_pages. This problem caused by lru empty but I admit it could work well if your patch applied.
So yours could help, too.

Quoted 3 lines

However I also think your patch is valuable. because while the task is sleeping in wait_iff_congested(), an another task may free some pages. thus, rebalance path should try to get free pages. iow, you makes sense.

Quoted 8 lines

Yes.
Off-topic.
I would like to move cond_resched below get_page_from_freelist in __alloc_pages_direct_reclaim. Otherwise, it is likely we can be stolen pages to other processes.
One more benefit is that if it's apparently OOM path(ie, did_some_progress = 0), we can reduce OOM kill latency due to remove unnecessary cond_resched.

I agree. Can you please mind to send a patch?

Quoted 13 lines

Quoted 3 lines

So, I'd like to propose to merge both your and my patch.

Recently, there was discussion on drain_all_pages with Wu. He saw much overhead in 8-core system, AFAIR.
I Cced Wu.

How about checking per-cpu before calling drain_all_pages() than unconditional calling?
if (per_cpu_ptr(zone->pageset, smp_processor_id()) drain_all_pages();

Quoted 3 lines

Of course, It can miss other CPU free pages. But above routine assume local cpu direct reclaim is successful but it failed by per-cpu. So I think it works.

Can you please tell me previous discussion url or mail subject? I mean, if it is costly and performance degression risk, we don't have to take my idea.

Thanks.

Quoted 2 lines

Thanks for good suggestion and Reviewed-by, KOSAKI.

Quoted 3 lines

Can you please tell me previous discussion url or mail subject? I mean, if it is costly and performance degression risk, we don't have to take my idea.

Yes. You could see it by https://lkml.org/lkml/2011/4/30/81.