Skip to contentSkip to conversation

Memory Problem in 2.4.9 ?

DISCUSSION Tommy Wu 3 participants 7 messages Aug 22, 2001 – Aug 23, 2001

lore
Share & tools
lore

Tommy Wu

author
#0 lore raw

Hi!

I've tried the patch in the kernel list. Got the result as following... This message for command:
dd if=/dev/zero of=test.dmp bs=1000k count=2500
on a PIII 1G SMP box with 1G RAM (HIGHMEM enabled)
kernel 2.4.9 with XFS filesystem patch.

Aug 22 11:51:04 standby kernel: __alloc_pages: 0-order allocation failed (gfp=0x30/1).
Aug 22 11:51:11 standby last message repeated 111 times
Aug 22 11:51:11 standby kernel: cation failed (gfp=0x30/1).
Aug 22 11:51:11 standby kernel: __alloc_pages: 0-order allocation failed (gfp=0x30/1).
Aug 22 11:51:11 standby last message repeated 281 times
Aug 22 11:51:17 standby kernel: cation failed (gfp=0x30/1).
Aug 22 11:51:17 standby kernel: __alloc_pages: 0-order allocation failed (gfp=0x30/1).
Aug 22 11:51:29 standby last message repeated 315 times
Aug 22 11:51:29 standby kernel: cation failed (gfp=0x30/1).
Aug 22 11:51:29 standby kernel: __alloc_pages: 0-order allocation failed (gfp=0x30/1).
Aug 22 11:51:29 standby last message repeated 281 times
Aug 22 11:52:21 standby last message repeated 43 times
Aug 22 11:52:21 standby kernel: cation failed (gfp=0x30/1).
Aug 22 11:52:21 standby kernel: __alloc_pages: 0-order allocation failed (gfp=0x30/1).
Aug 22 11:52:22 standby last message repeated 290 times
Signature · 6 lines
    Tommy Wu
    mailto:[email protected]
    http://www.teatime.com.tw/~tommy
    ICQ: 22766091
    Mobile Phone: +886 936 909490
    TeaTime BBS +886 2 31515964 24Hrs V.Everything
#1 lore raw
Quoted 9 lines · Tommy Wu

I've tried the patch in the kernel list. Got the result as following... This message for command:
dd if=/dev/zero of=test.dmp bs=1000k count=2500
on a PIII 1G SMP box with 1G RAM (HIGHMEM enabled)
kernel 2.4.9 with XFS filesystem patch.

Aug 22 11:51:04 standby kernel: __alloc_pages: 0-order allocation failed
(gfp=0x30/1).
Aug 22 11:51:11 standby last message repeated 111 times

OK, this is a straight-up design bug. Although this can also happen with normal memory, it's much more likely to happen with highmem because of heavy demand for bounce buffers while a process is in PF_MEMALLOC state. You can just turn off highmem and these messages will go away, or become so rare that you are unlikely to ever see one.

Now lets chase the real problem. The gfp=0x30 tells us the requestor is willing to wait (0x10) and that it is not allowed to do any io (0x40) or call ->writepage (0x80). (By process of elimination, it's a bounce buffer.) Furthermore, this is a recursive memory request (/1) so __alloc_pages won't call page_launder because that could hit another allocation request resulting in a fatal infinite recursion (note to self: why couldn't we call page_launder here, with NOIO?).

There are probably dirty pages in flight and __alloc_pages is allowed to wait for them, but it doesn't - it trys reclaim_page once (in __alloc_pages_limit), falls the rest of the way through __alloc_pages and gives up with NULL. This is clearly a bad thing because whoever wanted the page needs it to do writeout. Memory users are supposed to be able to tolerate alloc failure, but in a case like this, there isn't much choice other than to spin.

So what could we do better here? Well, obviously when there are writeout pages in flight, __alloc_pages should wait and not give up. Secondly, we should be sure that when writeout does complete, the newly freeable page is given to a PF_MEMALLOC waiter in preference to a normal user. We don't have mechanisms in place for doing either of those things right now, although some preliminary design ideas have been discussed. This gets way outside the bound of what we should be doing in 2.4, we will need such things as reservations (which Ben has done some work on) and orderly prioritization of requests in __alloc_pages, with explicit blocking for low priority requests.

What can we do right now? We could always just comment out the alloc failed message. The result will be a lot of busy waiting on dirty page writeout which will work but it will keep us from focussing on the question: how did we get so short of bounce buffers? Well, maybe we are submitting too much IO without intelligent throttling (/me waves at Ben). That sounds like the place to attack first.

--
Daniel

#2 lore raw
Quoted 43 lines · Daniel Phillips
Quoted 9 lines · Tommy Wu

I've tried the patch in the kernel list. Got the result as following... This message for command:
dd if=/dev/zero of=test.dmp bs=1000k count=2500
on a PIII 1G SMP box with 1G RAM (HIGHMEM enabled)
kernel 2.4.9 with XFS filesystem patch.

Aug 22 11:51:04 standby kernel: __alloc_pages: 0-order allocation failed
(gfp=0x30/1).
Aug 22 11:51:11 standby last message repeated 111 times

OK, this is a straight-up design bug. Although this can also happen with normal memory, it's much more likely to happen with highmem because of heavy demand for bounce buffers while a process is in PF_MEMALLOC state. You can just turn off highmem and these messages will go away, or become so rare that you are unlikely to ever see one.

Now lets chase the real problem. The gfp=0x30 tells us the requestor is willing to wait (0x10) and that it is not allowed to do any io (0x40) or call ->writepage (0x80). (By process of elimination, it's a bounce buffer.) Furthermore, this is a recursive memory request (/1) so __alloc_pages won't call page_launder because that could hit another allocation request resulting in a fatal infinite recursion (note to self: why couldn't we call page_launder here, with NOIO?).

There are probably dirty pages in flight and __alloc_pages is allowed to wait for them, but it doesn't - it trys reclaim_page once (in __alloc_pages_limit), falls the rest of the way through __alloc_pages and gives up with NULL. This is clearly a bad thing because whoever wanted the page needs it to do writeout. Memory users are supposed to be able to tolerate alloc failure, but in a case like this, there isn't much choice other than to spin.

So what could we do better here? Well, obviously when there are writeout pages in flight, __alloc_pages should wait and not give up. Secondly, we should be sure that when writeout does complete, the newly freeable page is given to a PF_MEMALLOC waiter in preference to a normal user. We don't have mechanisms in place for doing either of those things right now, although some preliminary design ideas have been discussed. This gets way outside the bound of what we should be doing in 2.4, we will need such things as reservations (which Ben has done some work on) and orderly prioritization of requests in __alloc_pages, with explicit blocking for low priority requests.

Quoted 6 lines

What can we do right now? We could always just comment out the alloc failed message. The result will be a lot of busy waiting on dirty page writeout which will work but it will keep us from focussing on the question: how did we get so short of bounce buffers? Well, maybe we are submitting too much IO without intelligent throttling (/me waves at Ben). That sounds like the place to attack first.

We can just wait on the writeout of lowmem buffers at page_launder() (which will not cause IO buffering since we are doing lowmem IO, duh), and then we are done.

Take a look at the patch I posted before (__GFP_NOBOUNCE).

#3 lore raw
Quoted 8 lines · Marcelo Tosatti
Quoted 6 lines · Daniel Phillips

What can we do right now? We could always just comment out the alloc failed message. The result will be a lot of busy waiting on dirty page writeout which will work but it will keep us from focussing on the question: how did we get so short of bounce buffers? Well, maybe we are submitting too much IO without intelligent throttling (/me waves at Ben). That sounds like the place to attack first.

Quoted 5 lines

We can just wait on the writeout of lowmem buffers at page_launder() (which will not cause IO buffering since we are doing lowmem IO, duh), and then we are done.

Take a look at the patch I posted before (__GFP_NOBOUNCE).

A little light reading for a Wednesday afternoon ;-)

Nice hack, way to go. So this will wait synchronously in try_to_free_buffers if we have to go around twice in alloc_bounce_page or alloc_bounce_bh (the latter eventually resulting in a page_alloc from kmem_cache grow).

What does SLAB_LEVEL_MASK do? Did you find out by hitting the BUG when you tried the patch? Anyway, it needs a comment.

I had in mind a completely different approach to try, using a semaphore to count bounce buffers, and block when they run out. Your patch fits the pattern of the current busy-waiting strategy much better. It's the right thing to do.

OK, race you to the next bug ;-)

--
Daniel

#4 lore raw
Quoted 13 lines · Daniel Phillips
Quoted 12 lines · Marcelo Tosatti
Quoted 6 lines · Daniel Phillips

What can we do right now? We could always just comment out the alloc failed message. The result will be a lot of busy waiting on dirty page writeout which will work but it will keep us from focussing on the question: how did we get so short of bounce buffers? Well, maybe we are submitting too much IO without intelligent throttling (/me waves at Ben). That sounds like the place to attack first.

We can just wait on the writeout of lowmem buffers at page_launder() (which will not cause IO buffering since we are doing lowmem IO, duh), and then we are done.

Quoted 7 lines
Quoted 1 line

Take a look at the patch I posted before (__GFP_NOBOUNCE).

A little light reading for a Wednesday afternoon ;-)

Nice hack, way to go. So this will wait synchronously in try_to_free_buffers if we have to go around twice in alloc_bounce_page or alloc_bounce_bh (the latter eventually resulting in a page_alloc from kmem_cache grow).

Not synchronously, no. It will just allow allocations trying to get memory for bounce buffering to block on lowmem IO.

With this behaviour, its safe to completly remove Ingo's emergency scheme.

Quoted 2 lines

What does SLAB_LEVEL_MASK do? Did you find out by hitting the BUG when you tried the patch? Anyway, it needs a comment.

SLAB_LEVEL_MASK is the mask for SLAB-valid allocations.

Quoted 6 lines

I had in mind a completely different approach to try, using a semaphore to count bounce buffers, and block when they run out. Your patch fits the pattern of the current busy-waiting strategy much better. It's the right thing to do.

OK, race you to the next bug ;-)

#5 lore raw
Quoted 17 lines · Marcelo Tosatti
Quoted 15 lines · Daniel Phillips
Quoted 13 lines · Marcelo Tosatti
Quoted 6 lines · Daniel Phillips

What can we do right now? We could always just comment out the alloc failed message. The result will be a lot of busy waiting on dirty page writeout which will work but it will keep us from focussing on the question: how did we get so short of bounce buffers? Well, maybe we are submitting too much IO without intelligent throttling (/me waves at Ben). That sounds like the place to attack first.

We can just wait on the writeout of lowmem buffers at page_launder() (which will not cause IO buffering since we are doing lowmem IO, duh), and then we are done.

Take a look at the patch I posted before (__GFP_NOBOUNCE).

Quoted 8 lines
Quoted 5 lines

A little light reading for a Wednesday afternoon ;-)

Nice hack, way to go. So this will wait synchronously in try_to_free_buffers if we have to go around twice in alloc_bounce_page or alloc_bounce_bh (the latter eventually resulting in a page_alloc from kmem_cache grow).

Not synchronously, no. It will just allow allocations trying to get memory for bounce buffering to block on lowmem IO.

Whoops, it's been a while since I read page_launder. Hey! Major cleanup. It's much easier to understand what it's doing now.

OK, sync_page_buffers no longer does what its name says, or implements what its comment says it does. Now the GFP_WAIT just means wait on already-locked buffers so that IO can be initiated. (By the way, there are bunch of comments in try_to_free_buffers that lie now.) OK, so the busy wait is implemented in alloc_bounce_page, and page_launder is just used to start IO, fine. Hmm, I think I will try my semaphore idea, not because you haven't solved the problem, but because I think a lot of CPU-wasting trips into page_launder could be eliminated. (A 2.5 thing of course.)

Quoted 1 line

With this behaviour, its safe to completly remove Ingo's emergency scheme.

Yes, so why don't you add that to your patch, and also the correction to the page->zone test and call it [PATCH]?

Quoted 4 lines
Quoted 2 lines

What does SLAB_LEVEL_MASK do? Did you find out by hitting the BUG when you tried the patch? Anyway, it needs a comment.

SLAB_LEVEL_MASK is the mask for SLAB-valid allocations.

And "LEVEL" means?

--
Daniel

#6 lore raw
Quoted 30 lines · Daniel Phillips
Quoted 17 lines · Marcelo Tosatti
Quoted 15 lines · Daniel Phillips
Quoted 13 lines · Marcelo Tosatti
Quoted 6 lines · Daniel Phillips

What can we do right now? We could always just comment out the alloc failed message. The result will be a lot of busy waiting on dirty page writeout which will work but it will keep us from focussing on the question: how did we get so short of bounce buffers? Well, maybe we are submitting too much IO without intelligent throttling (/me waves at Ben). That sounds like the place to attack first.

We can just wait on the writeout of lowmem buffers at page_launder() (which will not cause IO buffering since we are doing lowmem IO, duh), and then we are done.

Take a look at the patch I posted before (__GFP_NOBOUNCE).

Quoted 8 lines
Quoted 5 lines

A little light reading for a Wednesday afternoon ;-)

Nice hack, way to go. So this will wait synchronously in try_to_free_buffers if we have to go around twice in alloc_bounce_page or alloc_bounce_bh (the latter eventually resulting in a page_alloc from kmem_cache grow).

Not synchronously, no. It will just allow allocations trying to get memory for bounce buffering to block on lowmem IO.

Whoops, it's been a while since I read page_launder. Hey! Major cleanup. It's much easier to understand what it's doing now.

Quoted 8 lines

OK, sync_page_buffers no longer does what its name says, or implements what its comment says it does. Now the GFP_WAIT just means wait on already-locked buffers so that IO can be initiated. (By the way, there are bunch of comments in try_to_free_buffers that lie now.) OK, so the busy wait is implemented in alloc_bounce_page, and page_launder is just used to start IO, fine. Hmm, I think I will try my semaphore idea, not because you haven't solved the problem, but because I think a lot of CPU-wasting trips into page_launder could be eliminated. (A 2.5 thing of course.)

There is no real CPU-wasting trips into page_launder().

As soon as we find a dirty, lowmem, unlocked page with ->buffers, we block on IO there.

Quoted 4 lines
Quoted 1 line

With this behaviour, its safe to completly remove Ingo's emergency scheme.

Yes, so why don't you add that to your patch, and also the correction to the page->zone test and call it [PATCH]?

Will do it soon.

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

What does SLAB_LEVEL_MASK do? Did you find out by hitting the BUG when you tried the patch? Anyway, it needs a comment.

SLAB_LEVEL_MASK is the mask for SLAB-valid allocations.

And "LEVEL" means?

No idea. Nothing, probably.